github.com/thushan/olla

What's in this release

Olla is a high-performance proxy and load balancer for LLM infrastructure.

Quick Start

# Docker
docker pull ghcr.io/thushan/olla:v0.0.29

# Binary (see assets below)
./olla --config config.yaml

What's New in v0.0.29

Admin Dashboard

• New embedded, read-only admin dashboard at /internal/ui/ - Overview (fleet status, success rate, latency, live requests-per-second sparkline), Endpoints (per-endpoint health, priority, latency, model count) and Models (inventory grouped by family, with hosting endpoints). Built in Svelte 5 + TailwindCSS, polled (not pushed) against the existing /internal/status* JSON, and served from the same listener as the proxy - no second port (#205).
• The dashboard has no authentication. Access is controlled by a new dashboard.access_policy config block (allowed_cidrs + allowed_hosts), loopback-only by default. The published Docker image ships with allowed_cidrs pre-widened to the RFC1918 ranges so docker run -p 40114:40114 ghcr.io/thushan/olla:latest gives you a working dashboard with no config mount - which also means the container is reachable from anyone else on the same LAN through that published port (#213). See the Admin Dashboard docs for the full security model.
• A binary built without make build-web (e.g. plain go build or go install) now logs a clear startup warning and serves 503 at /internal/ui/ instead of a silent placeholder (#213).

Observability

• Native GET /internal/metrics in Prometheus text format, built from the same data as /internal/status and /internal/stats/models - no external exporter needed for core proxy monitoring. Thanks to @Puupuls for the contribution (#188).
• /internal/status, /internal/status/endpoints and /internal/status/models now emit a weak ETag, so polling clients (including the new dashboard) can use If-None-Match and get 304s instead of re-fetching the full payload (#205, #213).
• Sticky-session and routing decisions are now in the structured "Request completed" log line (sticky_outcome, routing strategy/action/reason, provider_model, translator fallback reason), not just response headers. Addresses issue #178 (#182).
• Successful proxy requests log at Info again in the access log; a prior regression had silently demoted them to Debug (#211).

Reliability

• A request landing on an endpoint whose circuit breaker is open now fails over to the next available endpoint instead of failing outright. The endpoint is dropped from that request's candidate list only - its persisted health is left alone, since an open breaker already reflects accumulated failure state and demoting health on top of it isn't the retry path's job (#213).
• Strict routing no longer lets an unroutable model_aliases entry silently proxy to the wrong backend: a request for an alias whose target model exists on no endpoint now fails fast with 404/503 and routing_action: rejected, matching how unknown models are already handled. Reported by @skaravos (#197, issue #191).
• config/models.yaml now actually parses. It used \d inside double-quoted YAML scalars, which is invalid YAML, so the file has never loaded on any install - the failure was swallowed and Olla silently fell back to embedded defaults, discarding any customisation with no diagnostic. Reported by @billford (#206, issue #204).
• New --validate-config flag checks configuration and provider profiles without starting the server, with a pass/warn/fail report and exit codes (#210).
• logging.level in config is now actually applied to the runtime logger, instead of being parsed and ignored (#209).

Model Catalog

• Refreshed the built-in model catalog for 2026: recognises more families and variants (GPT-OSS, GLM, Kimi, Granite, Nemotron, EXAONE, Hunyuan, MiniMax, OLMo, InternLM, SmolLM, Command-R, Forge) and MXFP4/NVFP4 quantisation (#207).
• Wired model_extraction.family_aliases and special_rules.preserve_family into the family-extraction pipeline - both were parsed but previously had no effect. Fixes Kimi-K2 being misclassified under deepseek (shared GGUF architecture lineage) (#208).

Security & Startup

• An endpoint URL with embedded user:pass@host credentials now fails startup instead of loading silently and leaking the credentials into every status/dashboard JSON response. The boot error rewrites the URL into a ready-to-paste auth: block with placeholders, so the fix doesn't require re-typing the real credentials (#213).
• The id and url fields on /internal/status and /internal/status/endpoints are now derived from a sanitised URL (userinfo, query and fragment stripped) rather than echoing the raw configured URL (#213).
• /internal/status no longer reports "status": "critical" on a healthy fresh boot with no traffic yet. A new system.has_traffic boolean lets clients branch on the no-traffic state directly instead of parsing the success_rate string (#213).

Docker & Release

• Streamlined the container image and switched to .dockerignore for a smaller, more predictable build context (#213).
• Added a release dry-run validation step to the release pipeline (#213).

Breaking Changes

• Userinfo URLs: - url: "http://user:pass@host:8080" now fails startup. Move credentials into an auth: block:

  - url: "http://host:8080"
    auth:
      type: basic
      username: user
      password: pass

• Status id/url fields: now derived from a sanitised URL rather than the raw config value. Bookmarked dashboard deep-links to a specific endpoint will change once after upgrading. Clients comparing url for identity should switch to id.
• Zero-traffic status: a fresh boot with all endpoints healthy no longer reports "critical". Check system.has_traffic if you branch on traffic state.
• Strict routing on unroutable aliases: /olla/proxy/ with zero healthy endpoints now returns 503 instead of 502. Update any monitor keyed on that status code.
• owned_by in converted model listings: the shared organisation-extraction logic used by the vLLM, vLLM-MLX, SGLang, llama.cpp, LMDeploy and Lemonade converters now also matches a hyphenated leading segment (e.g. Qwen2.5-7B → owned_by: "Qwen2.5") rather than falling through to a generic default. Treat owned_by as a best-effort label, not a canonical identifier.

Acknowledgements

Thanks to everyone who contributed to this release:

• @Puupuls - contributed the native GET /internal/metrics Prometheus endpoint (#188).
• @skaravos - reported the strict-routing alias bypass (issue #191).
• @billford - reported the config/models.yaml parse failure (issue #204).
• @sg-shag - reported the missing sticky-session/routing decision logging (issue #178).

Full Changelog: v0.0.28...v0.0.29

Documentation: thushan.github.io/olla | Issues: github.com/thushan/olla/issues

Olla is a high-performance proxy and load balancer for LLM infrastructure.

Quick Start

# Docker
docker pull ghcr.io/thushan/olla:v0.0.28

# Binary (see assets below)
./olla --config config.yaml

What's New in v0.0.28

This is a huge release and contains a lot of new exciting changes and bugfixes!

New Backends & Compatibility

• Native oMLX support - added the oMLX runtime as a first-class backend with native Anthropic Messages API passthrough (#167).
• Docker Model Runner Anthropic passthrough - passthrough now resolves the native messages path from the backend profile (anthropic_support.messages_path), fixing 404s against DMR's /anthropic/v1/messages (#171).
• Lemonade models are now routable - Lemonade's downloaded flag is mapped to available state, so downloaded models route correctly instead of being treated as unhealthy. Thanks to @matthewjhunter for the report and fix (#161, issue #160).
• OpenAI alias fixes - type: "openai" now resolves model listings correctly via /olla/openai/v1/models; removed the duplicate openai.yaml profile that caused non-deterministic route registration. openai-compatible is the canonical value,
  openai remains an accepted alias. Thanks to @petersimmons1972 for the report (#151, issue #148).
• OpenAI-compatible backends now surface maximum context-length information (#167).

Authentication & Security

Highly requested feature finally implemented!

• Per-endpoint authentication for local backends - bearer, api_key and basic auth, with credentials from inline strings, ${ENV_VAR}, or _file siblings for Docker/k8s. Works with vLLM/llama.cpp/LiteLLM --api-key or any bearer-auth reverse
  proxy. Thanks to @ShubhamTiwary914 for the report (#146, issue #132).
• 401/403 health probes now report config_error instead of dead; 429 honours Retry-After without tripping the circuit breaker; POST retries are skipped once response bytes have flushed (no double billing on mid-stream resets).
• Capped upstream error bodies at 1 MiB and enforced max_body_size on non-proxy and translator routes (including chunked bodies) (#173).
• Strip inbound X-Olla-* headers from upstream responses (anti-spoofing), sanitise X-Request-ID to prevent log injection, and stop leaking the auth scheme in status JSON (#173).

CORS

• CORS support (opt-in, via rs/cors) for browser-based clients like OpenWebUI and dashboards. Off by default; auto-exposes the X-Olla-* header set. Thanks to @ccsmart for the request (#159, issue #156).

Anthropic / Translation

• Reasoning → thinking translation - OpenAI reasoning/reasoning_content now maps to Anthropic thinking blocks in both streaming and non-streaming responses (#176).
• Hardened streaming translation: preserves final usage chunks, synthesises fallback output_tokens, keeps same-chunk reasoning/content/tool-call deltas intact, and fixes repeated/interleaved tool-call handling (#176).
• More reliable translated SSE: emits valid SSE error events after response commit, with proper Flush/Unwrap on wrapped writers (#176).
• Lenient Anthropic parsing - unknown/experimental Anthropic request fields (e.g. context_management) are now ignored during translation and forwarded unchanged in passthrough mode. Thanks to @RenxuLogan for the report (#157, issue #154).

Configuration Tunables

• New tunables, all zero-value-means-default and OLLA_-overridable (#164, #165):
• server.read_header_timeout (default 10s) - guards against Slowloris-style slow-header attacks.
• proxy.response_header_timeout (default 30s) - raise for on-demand model loaders (e.g. Lemonade) that would otherwise abort cold starts at 30s. Now honoured by both Olla and Sherpa. Thanks to @matthewjhunter (#164, issue #163).
• proxy.connection_keep_alive (default 30s) and proxy.tls_handshake_timeout (default 10s).
• model_registry.unification.cleanup_interval / stale_threshold are now honoured (previously silently ignored, hard-coded to 5m) (#171).

Reliability & Performance

• Olla config is now an atomic.Pointer, snapshotted once per request/stream, so a hot-reload can't hand an in-flight request a torn config (#173, #176).
• Endpoint pools and circuit breakers use LoadOrCompute, eliminating racing transport rebuilds on first use (#173).
• Cleanup loops survive per-tick panics; fixed event-bus shutdown and health-checker double-stop races; proxy engine cleanup now runs on shutdown (#176).
• litepool.Get returns errors instead of panicking (handles nil and typed-nil factory results) (#176).
• Hot-path tuning: typed-struct streaming chunk parsing and pooled SSE encoding buffers (#176).
• Removed dead code in the Olla engine (unused methods/pools, a forced runtime.GC(), an unread request counter) (#173).

Behaviour Changes to be aware of

• /internal/status/models now returns endpoint names for every entry (was inconsistently URL-then-names). Consumers reading endpoints[0] as a URL must switch to names (#173).
• Upstream X-Olla-* response headers are dropped, so a chained Olla-in-front-of-Olla setup loses the inner instance's headers (#173).
• Default proxy engine is now Olla and the default load balancer is least-connections (#172).

Tooling & Docs

• New validation harness (/olla-validate --quick / --nightly) with a local multi-protocol mock backend and fault injection - no Docker, CI-safe (#174).
• Standardised GitHub issue/PR templates (#170).
• Documentation refresh aligning all guides, headers and config examples with the codebase (#172).
• Fixed /version self-path reporting (previously reported a 404'ing /internal/version) (#171).

Acknowledgements

Thanks to everyone who reported issues and contributed to this release:

• @matthewjhunter - contributed the Lemonade routing fix (#161, issue #160) and the configurable response_header_timeout (#164, issue #163).
• @ShubhamTiwary914 - reported the need for auth against local backends (#132), which drove per-endpoint authentication.
• @petersimmons1972 - reported empty model listings for type: "openai" endpoints (#148).
• @RenxuLogan - reported newer Anthropic request fields breaking translation (#154), driving lenient parsing.
• @ccsmart - requested CORS support for browser-based clients (#156).

What's Changed Summary

• fix: openai endpoint type returning empty model listings by @thushan in #151
• fix: be lenient for anthropic parsing by @thushan in #157
• feat: Authentication for local backends by @thushan in #146
• feat: CORS implementation by @thushan in #159
• fix: route Lemonade models by mapping downloaded to available state by @matthewjhunter in #161
• fix: make proxy response_header_timeout configurable by @matthewjhunter in #164
• feat: configuration tunables by @thushan in #165
• feat: backend omlx by @thushan in #167
• ci: improve issues templates etc. by @thushan in #170
• fix: anthropic passthrough path for Docker Model Runner + align config defaults by @thushan in #171
• align documentation with the codebase for v0.0.28 by @thushan in #172
• feat: olla hardening June 2026 by @thushan in #173
• feat: olla mocking skills for claude by @thushan in #174
• rollup: June 2026 by @thushan in #176

New Contributors

• @matthewjhunter made their first contribution in #161

Full Changelog: v0.0.27...v0.0.28

Documentation: thushan.github.io/olla | Issues: github.com/thushan/olla/issues

What's in this release

We've added native support for LMDeploy after so long as well as bugfixes for sticky sessions thanks to @lbatalha.

Quick Start

# Docker
docker pull ghcr.io/thushan/olla:v0.0.27

# Binary (see assets below)
./olla --config config.yaml

Changelog

Other

• 4ed9cf0: Bump github.com/puzpuzpuz/xsync/v4 from 4.4.0 to 4.5.0 (@dependabot[bot])
• b900a20: add aimock test harness for sticky sessions (@thushan)
• f350dcd: add language tags to skill code fences (@thushan)
• 249e915: add lmdeploy backend docs (@thushan)
• e52b561: add lmdeploy model converter and wire into factory and routes (@thushan)
• 364154e: add lmdeploy profile (@thushan)
• a06a3a9: add lmdeploy provider constants (@thushan)
• 252d0ea: add lmdeploy response parser (@thushan)
• 3fa1875: assert backend marker is non-empty in sticky tests (@thushan)
• 45f9865: clarify provider prefix helpers return no trailing slash (@thushan)
• 19b712b: doc updates and lmdeploy TPS is still in the works. (@thushan)
• ab4a786: fix lmdeploy struct alignment (@thushan)
• a57501b: fix prefix_hash fallback for empty messages array (@thushan)
• 8510341: fix sticky diversity check to ignore failed requests (@thushan)
• 3055480: fix sticky sessions for provider-scoped routes (#139) (@thushan)
• 95022d1: link lmdeploy from backend overview docs (@thushan)
• 4ac53f2: pin aimock image digest for reproducible tests (@thushan)
• bbdd5f0: readme update for native support for LMDeploy (@thushan)
• 85495d5: revert anthropic_support default + fix stale comment (@thushan)
• 7985c7c: stabilise sticky harness turn-3 diversity (@thushan)
• ac2386f: use docker compose --wait instead of custom poll (@thushan)

Documentation: thushan.github.io/olla | Issues: github.com/thushan/olla/issues

What's in this release

This is a bugfix release that addresses SSL connection issues due to mis-handling of the Host header.

Quick Start

# Docker
docker pull ghcr.io/thushan/olla:v0.0.26

# Binary (see assets below)
./olla --config config.yaml

Changelog

Other

• 6896157: Bump actions/upload-pages-artifact from 4 to 5 (@dependabot[bot])
• 0d3ff86: doc updates (@thushan)
• 50287bf: lets you build docker image locally without goreleaser (@thushan)
• 9505fba: make docker builds portable across arm and amd (@thushan)

Documentation: thushan.github.io/olla | Issues: github.com/thushan/olla/issues

What's in this release

Olla is a high-performance proxy and load balancer for LLM infrastructure.

Quick Start

# Docker
docker pull ghcr.io/thushan/olla:v0.0.25

# Binary (see assets below)
./olla --config config.yaml

Release Highlights

Model Aliasing

Thanks to @dnnspaul for contributing the Model Aliasing feature to Olla to alias models easily via the configuration.

Sticky Sessions

We've now got a way of having sticky sessions in Olla to help keep requests aligned to KV Caches across multiple endpoints, taken from the working implementation in TensorFoundry's FoundryOS.

Bugfixes and Chores

Lots of bugfixes and chores from March & April.

Changelog

Features

• 8378bff: feat: add model alias validation, test coverage, and byte-preserving JSON rewrite (@dnnspaul)
• pr: sticky-sessions sticky sessions implementation (@thushan)

Bug Fixes

• 33307eb: fix(inspector): copy buffer bytes before pool return to avoid aliasing (@dnnspaul)
• c54eea9: fix(inspector): incremental scan with token-level skipping for field-order independence (@dnnspaul)
• f7222c9: fix(inspector): replace io.NopCloser with readCloser to preserve body Close delegation (@dnnspaul)
• 2bbb9f7: fix(inspector): restore body on error and fix decoder state in extractTopLevelModelField (@dnnspaul)
• ed1609a: fix: address CodegRabbit review issues (@thushan)
• 8a28dbf: fix: extract model name from large requests via streaming JSON prefix scan (@dnnspaul)
• d923536: fix: propagate model alias rewrite map to translation hanlder (@thushan)
• 96b67eb: fix: replace regex model rewrite with json.Decoder token scanner (@dnnspaul)

Other

• f4104ef: + documentation (@dnnspaul)
• df441b3: - OLLA-284: move SetPurgeDeadEndpoints registration from app.go wiring time into applyStickySessions() (@thushan)
• 8b9e939: CI fix for actions etc. (@thushan)
• 7e426e9: Fix latent rarce on purgedead & replace TTL sleep with poll loop (@thushan)
• 5268939: betteralign adjustments (@dnnspaul)
• 3cbef31: bump x/sync to 0.20.0 (@thushan)
• 414e5a9: bump x/time to 0.15.0 (@thushan)
• 83712ca: concise (@thushan)
• d15a63d: doc updates (@thushan)
• afb0f25: fix leaky test (@thushan)
• 047cc92: fix test failures from typed model key and retry string trim (@thushan)
• 44999cf: fix ttlcache == 0 (@thushan)
• ca899c5: guard Cleanup against double-invoke (@thushan)
• 307b9c4: guard StopChecking against concurrent double-invoke (@thushan)
• a99a8bb: implementation(PR-98): model aliases (@dnnspaul)
• efb02fa: initial sticky sessions work (@thushan)
• 17d167f: lint test (@thushan)
• f717468: refactor(inspector): clarify pool-safety asymmetry and add capability regression guard (@dnnspaul)
• 1c209c0: refactor: address PR review feedback (round 2) for model aliases (@dnnspaul)
• 41b544e: refactor: address PR review feedback for model aliases (@dnnspaul)
• 4bbe95b: reference updates (@thushan)
• 6b9e2d0: remove dead proxyToSingleEndpointLegacy (@thushan)
• 29b6bca: remove dead responsePool (@thushan)
• 9c0debc: reorder Service fields for betteralign (@thushan)
• 6de613d: revert x/sync bump, needs go 1.25 (@thushan)
• ec5d03f: revert x/time bump, needs go 1.25 (@thushan)
• da5d8bb: tighten connection error string fallback in retry (@thushan)
• 01500b7: update coderabbit commnts (@thushan)
• 94c1784: update readme (@thushan)
• ce4aaaa: update readme (@thushan)
• 8442085: update version signature to be a bit more robust (@thushan)
• c2d9af4: use typed context key for model (@thushan)
• c9f2223: wrong template . (@thushan)

Documentation: thushan.github.io/olla | Issues: github.com/thushan/olla/issues