NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
Go modules, #443 by repository stars
Latest stable
v0.120.23
Last release today
21 Nov 2026
Ships on a steady schedule
a new release about every 8 days
Nearly every release is documented
notes for 58 of the last 60 stable releases
Nothing withdrawn
no release was ever pulled
4 years old
2616 releases · first in 2023
Drag across the ruler to compare, or tap a release to open it.
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Your coding agent can read these notes before it upgrades. Set up the MCP server →
fix(alerts): select failure alerts by run finish time
fix(alerts): select failure alerts by run finish time
re-stamp finished_at when a replayed run fails again
read failure alert candidates from the primary
build failure-alert index concurrently on every partition
drop IF NOT EXISTS from finished_at column add
fix(auth): rotate session ID on login and password change
fix(auth): rotate session ID on login and password change
fix(auth): keep the cookie when a save hits a revoked session
A stale in-flight request would otherwise clear the cookie a concurrent
login or password change just rotated. The next request mints a fresh
ID from a dead cookie anyway.
Nothing published for this version
Nothing published for this version
Nothing published for this version
chore: add audit log tables to oss
chore: add audit log tables to oss (#5389)
chore: add audit log tables to oss
chore: add audit log tables to oss (#5389)
Nothing published for this version
fix: use actual partition bound for finding partitions before a date …
fix: use actual partition bound for finding partitions before a date …
refactor: start cleaning up hypertable removal
refactor: start cleaning up hypertable removal
fix: simplify partition creation
chore: simplify partitioning query
fix: comment
refactor: start cleaning up hypertable removal
refactor: start cleaning up hypertable removal
fix: simplify partition creation
chore: simplify partitioning query
fix: comment
perf(dispatcher): read tasks, inputs and runtimes in one statement before sending
ListTasksForDispatch returns each assigned task with its current runtime row and
the payload row holding its input, located through read_payloads, so the
dispatcher builds an action from one round trip instead of three. A task whose
runtime row is gone is not returned and not sent. A failed read still requeues
the requested tasks through a bounded recovery lookup. The scheduler no longer
publishes an empty task-assigned-bulk message for locally delivered items.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
schedule-step-runs looked up the dispatcher of every assigned worker on
each bulk of assignments (0.74 ms of the 0.86 ms step on the dispatcher-side
trace). The scheduler now keeps a 10000-entry, one-second TTL cache from
worker id to dispatcher id in front of GetDispatcherIdsForWorkers, consults
it first and queries only the worker ids it holds no entry for, adding what
comes back. There is no invalidation, matching the other repository caches:
a worker that moves dispatchers is seen within one second, the same window
as its liveness check. Workers without a dispatcher are not cached and are
queried again. The cancellation and dead-letter paths keep querying.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
ListTasksForDispatch stayed on a custom plan for every call under the pgx
default of prepared statements and plan_cache_mode=auto: with the id array
unknown the generic plan is costed at ten elements (354 against the custom
plan's 251 on the bench database), so Postgres never adopted it and every
execution paid 0.9 to 2.2 ms of planning for 0.1 to 0.3 ms of work.
The statement now reads ONE task by scalar parameters (tenant_id, task id):
the task row, its runtime row joined on the primary key, and the payload rows
read_payloads returns for one-element arrays built from that row. Nothing in
its estimate depends on a parameter value, so the generic plan costs the same
as the custom plan and is adopted on the sixth execution. The repository
queues one statement per task id in a single pgx batch
(ListTasksForDispatchBatch, hand-written over the generated constant and row
type), so a bulk of assignments is still one round trip. read_payloads
remains the only place with payload lookup rules; the Go side
(tasksForDispatchFromRows, Resolve, the recovery requeue) is unchanged.
Measured on a scratch Postgres 15.6 (three daily partitions of v1_task and
v1_payload, 30k tasks, 60k payload rows, 30k runtime rows, ANALYZEd), one
task id per execution, EXPLAIN (ANALYZE) EXECUTE under plan_cache_mode=auto;
"after 6" is the mode of the sixth and seventh execution, planning and
execution are the seventh's:
shape generic custom after 6 plan ms exec ms
current (tasks CTE x3, array param) 471.12 317.37 custom 0.836 0.130
(a) NOT MATERIALIZED tasks CTE, pkey runtime 642.55 126.54 custom 0.826 0.111
(a') MATERIALIZED variant of the same 699.40 333.66 custom 0.687 0.095
(b) per-task LATERAL read_payloads, array 464.87 100.81 custom 0.670 0.085
(c) per-task LATERAL, scalar params 84.23 84.23 generic 0.006 0.070
(c') shipped: task CTE, scalar params 202.91 202.91 generic 0.011 0.103
(c) is the cheapest plan but sqlc types the columns of a LEFT JOIN LATERAL
function as NOT NULL, which would fail to scan a task without a payload row;
(c') keeps the payload lookup in a CTE joined with ON true, which sqlc types
as nullable, and reaches the same mode. Cold connection, fresh session, per
round over seven rounds (three repetitions): main's three statements planned
2.7 to 4.2 ms in round 1 and 0.6 to 0.9 ms in rounds 2 to 6; the shipped
statement planned 3.0 to 3.4 ms in round 1, 0.6 to 0.7 ms in rounds 2 to 5,
and 0.007 ms from round 7, with execution 0.07 to 0.10 ms.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
The dispatch read runs one statement per task, so tasks sharing a
payload_reference_id each return their own copy of the owner's payload
row. tasksForDispatchFromRows appended every copy, and Resolve then
mapped every referrer onto every copy, which is quadratic in the number
of tasks sharing a row. Keep each payload row once, keyed by
(tenant_id, id, inserted_at, type).
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
The scheduler's worker to dispatcher cache expired entries on a one
second TTL alone, so every cached worker was re-queried once a second
and a worker that reconnected to another dispatcher was routed to the
old one until its entry expired.
The dispatcher now publishes a worker-dispatcher-changed message to the
tenant's scheduler partition right after it records itself on a
reconnecting worker's row, carrying the worker id and the new
dispatcher id. Both Listen paths share that step through claimWorker,
which leaves a worker already on this dispatcher alone. The scheduler
routes the message into the cache with set, so the entry is replaced
on the event without a query.
With updates event-driven, the TTL moves from 1 s to 5 s, the worker
liveness window; it remains the safety net for a lost publish. An
older scheduler logs the unknown message id and drops it, and an older
dispatcher simply never sends it, so mixed-version fleets fall back to
the TTL.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
The dispatcher's pre-send read queued one scalar-parameter statement per
task in a pgx batch so that each execution planned generically; Postgres
runs the queued statements serially, so a bulk of N assignments cost N
executions. ListTasksForDispatch is one statement again, taking the id
array, and still settles on the generic plan under the pgx default of
prepared statements with plan_cache_mode=auto.
Why id = ANY($ids) and a plain unnest($ids) cannot: under auto Postgres
keeps the custom plan while it costs lower than the generic one. The custom
plan sees the array as a constant and knows its length (estimate_array_length
for ANY, array_unnest_support for unnest), while the generic plan assumes
ten elements, so for short arrays the custom plan always costs less and
every execution is planned again. The statement now reads the ids through
unnest((SELECT $ids::bigint[])): the scalar subquery becomes an initplan
whose output both plans estimate at ten rows, so the two costs coincide and
the generic plan is adopted on the sixth execution. The per-task payload
lookup is a LEFT JOIN LATERAL of read_payloads with one-element arrays, the
runtime match is an inner join on the primary key, and the plan probes
v1_task_runtime and v1_task by index per id with runtime partition pruning.
sqlc types the columns of a LEFT JOIN LATERAL function as NOT NULL (also
when wrapped in a subselect), which would fail to scan a task without a
payload row, so the statement, its nullable row type and the reader are
hand-written in tasks-overwrite.go like CreateTasks and the query leaves
tasks.sql. tasksForDispatchFromRows (with the payload-row dedupe), Resolve,
the dispatcher and the dispatcher id cache are unchanged.
Measured on a scratch Postgres 15.6 (three daily partitions of v1_task and
v1_payload, 30k tasks, 60k payload rows, 30k runtime rows, ANALYZEd),
PREPARE (uuid, bigint[]) and EXPLAIN (ANALYZE) EXECUTE under
plan_cache_mode=auto; "after 6" is the plan of the sixth and seventh
execution, planning and execution are the seventh's, one execution for all
ids:
shape ids generic custom after 6 plan ms exec ms
0 main's CTEs, id = ANY($ids) 1 471.29 317.53 custom 1.175 0.166
20 471.29 631.79 generic 0.010 1.635
100 471.29 1388.31 generic 0.017 30.610
A unnest($ids), LATERAL read_payloads 1 145.45 60.44 custom 0.707 0.071
20 145.45 239.91 generic 0.013 0.305
100 145.45 871.57 generic 0.018 0.992
B unnest($ids), aggregated payloads CTE 1 605.28 317.13 custom 0.891 0.140
20 605.28 914.78 generic 0.009 1.578
100 605.28 2496.53 generic 0.028 30.912
C unnest WITH ORDINALITY, LATERAL 1 145.38 60.42 custom 0.743 0.074
20 145.38 239.79 generic 0.007 0.261
100 145.38 871.05 generic 0.011 0.898
D unnest((SELECT $ids)) CTE, LATERAL 1 145.46 145.46 generic 0.006 0.075
20 145.46 145.46 generic 0.007 0.260
100 145.46 145.46 generic 0.013 0.914
D2 shipped: D without the CTEs 1 145.39 145.39 generic 0.010 0.085
20 145.39 145.39 generic 0.012 0.270
100 145.39 145.39 generic 0.017 0.950
D3 D2 with LATERAL (SELECT * FROM ...) 1 145.39 145.39 generic 0.014 0.085
20 145.39 145.39 generic 0.017 0.292
100 145.39 145.39 generic 0.018 1.014
E unnest((SELECT $ids)), payloads CTE 1 605.29 605.29 generic 0.006 0.109
20 605.29 605.29 generic 0.010 1.574
100 605.29 605.29 generic 0.014 30.427
F id = ANY((SELECT $ids)), LATERAL 1 464.91 464.91 generic 0.022 0.199
20 464.91 464.91 generic 0.011 0.263
100 464.91 464.91 generic 0.014 0.922
G id = ANY((SELECT $ids)), payloads CTE 1 699.58 699.58 generic 0.034 0.239
20 699.58 699.58 generic 0.011 2.017
100 699.58 699.58 generic 0.055 35.494
The aggregated-CTE shapes (0, B, E, G) go generic for 20 or more ids but
join the tasks to the payload CTE with an OR condition that runs in N x M,
30 ms at 100 ids. The LATERAL shapes stay at one index probe per task. D2
is D without the CTEs so sqlc.embed resolves against the table; its plan is
identical. Cold connection, fresh session, planning ms per round over seven
rounds with one id (three repetitions): 6.6/3.9/3.2 in round 1, 0.75 to
1.23 in rounds 2 to 6, 0.035/0.018/0.011 from round 7, with execution 0.10
to 0.61 ms. Through pgx on one connection, pg_prepared_statements shows
five custom plans and then generic plans from the sixth call on.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Drops the dispatcher unit tests and the repository dispatch read test, which the end
to end suites cover, removes a narrating comment, and documents the worker dispatcher
change notification.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
A batch start that reaches a dispatcher the worker has left now takes the same
retry path as single assignments, so the scheduler reassigns the items instead
of leaving them to their step timeout. The dispatch read marks a present payload
row by its id and reads the external id directly.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
fix: internal retries piling up
fix: add query to release operator on shutdown and wire it up
fix: mark dispatcher draining so we don't have a shutting-down dispatcher claiming an operator
fix(dispatcher): carry the attempt's retry count so failed-read recovery never fails a newer attempt
The task-assigned-bulk message now carries the retry count of every task it
assigns, keyed by worker and task id. The dispatcher requeues a task after a
failed dispatch read only when its runtime row names the same worker and the
same retry count as the message. A task whose retry count the message does not
carry is left to the timeout path.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
fix: simplify a bunch after alexander's changes
fix: embed structs to clean up the code a bit
refactor: simplify, fix a couple edge cases, rm unused query
fix: simplify, clean up
fix: conflicts
fix: restore some stuff
fix: simplify more
Co-authored-by: Alexander Belanger alexander@hatchet.run
Co-authored-by: Claude Fable 5.1 noreply@anthropic.com
Nothing published for this version
feat(durable-streams): Go, Python, Ruby SDKs
feat(durable-streams): Go, Python, Ruby SDKs (#5348)
Nothing published for this version
Nothing published for this version
feat: add optional read-only API tokens
Read-only tokens are denied on gRPC and authorized as viewers on the REST API, so a credential can inspect a tenant without connecting a worker.
fix lint
fix em dashes
Co-authored-by: Julius Park juliusgeo@gmail.com
feat: add union queries and read goose version for cutover time
feat: read out of new aggs
feat: add union queries and read goose version for cutover time
fix: one day offset
fix: simplify a bunch
fix: naming
fix: simplify
fix: simplify more
fix: simplify diff
fix: date rounding
https://github.com/hatchet-dev/hatchet/discussions/5358#discussioncomment-DC_kwDOK6NWss4BJcSm
https://github.com/hatchet-dev/hatchet/discussions/5358#discussioncomment-DC_kwDOK6NWss4BJcSm
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
fix: migration deadlocks
fix: migration deadlocks (#5357)
feat: create new tables for aggregates
feat: create new tables for aggregates
fix: exclude dag orchestrators from task events
feat: add aggregation logic
feat: add compaction queries
chore: add some tests
chore: rm more slop comments
chore: fix noisy stall invocation error on eviction
chore: fix noisy stall invocation error on eviction
fix: noisy error on shutdown
fix: one more
feat: add a Concurrency page to the dashboard
Lists every concurrency strategy in the tenant (task, workflow and shared
scopes) with live key, running and queued counts, and shows a strategy's
keys in the side panel.
chore: regenerate SDK REST clients for the concurrency endpoints
fix: declare x-payloads on the concurrency operations
fix: address review feedback on the concurrency overview
fix: use the suggested empty state and round-robin copy
review
remove llm useless comments
fix test
remove limit strategy tooltips
remove concurrency page description
remove keys footer info tooltip
remove filters and use neutral limit strategy colors
show strategy details in the keys drawer
regenerate typescript fetch client
fix keys refresh query key
hide the concurrency page behind a feature flag
use the tenant pool handle for concurrency queries
Nothing published for this version
fix(engine): remove advisory lock on DAG replays using the DAG operat…
fix(engine): remove advisory lock on DAG replays using the DAG operat…
fix: one more shutdown issue chore: changelog
fix: one more shutdown issue
chore: changelog
Nothing published for this version
fix: partition pruning in status updates
fix: partition pruning in status updates
fix: geq
fix: add more pruning to dag query while we're at it
chore: lint
Nothing published for this version
Nothing published for this version
Any authenticated operation can now be throttled, so the spec declares it: a shared TooManyRequests response (APIErrors body, Retry-After header) on e
Any authenticated operation can now be throttled, so the spec declares it: a shared TooManyRequests response (APIErrors body, Retry-After header) on every operation that requires authentication. The two operations that answer 429 for a tenant quota keep their inline response, which has no Retry-After.
The API and webhook rate limiters set Retry-After to the seconds until their next token, so their 429s match the declared response.
The chore: regenerate API code commit is regenerated code only (Go server, Go client, frontend client). The chore(sdks) commit regenerates the Python, TypeScript and Ruby REST clients and bumps each SDK's patch version.
Nothing published for this version
Nothing published for this version
Nothing published for this version
fix: deadlocks
fix: deadlocks (#5337)
chore: add env var for allowing reassigns on unacked tasks
chore: add env var for allowing reassigns on unacked tasks
fix: only write assign times when acks are enabled
chore: simplify query
chore: document
fix: only look up durable inv counts if any task is durable
refactor: sql cleanup
fix: dont need type cast
fix: gen
fix: error message
fix: query
fix: naming
chore: rm comments
chore: add comment to query
fix: greptile comments
fix: only look up durable inv counts if any task is durable
fix: one more
fix: build issue
fix: queries on large table
fix: comments
fix: query optim
fix: add worker id to ack query
fix: wiring
chore: revert
fix: merge
fix(engine): durable invocation hangs until eviction on stream discon…
fix(engine): durable invocation hangs until eviction on stream discon…
chore: rm olap cutover job chore: remove a ton of sql chore: rm another query
chore: rm olap cutover job
chore: remove a ton of sql
chore: rm another query
Hatchet v0.114.2 adds rate limit pausing, RetryAfter errors, and FIPS images. It also includes a set of security fixes and a large batch of DAG, sched…
Hatchet v0.114.2 adds rate limit pausing, RetryAfter errors, and FIPS images. It also includes a set of security fixes and a large batch of DAG, scheduler, and database fixes.
RetryAfter errors: a task can return a RetryAfterError to choose when its next retry runs. See Retry Policies (#5162).-fips suffix, built against the validated Go Cryptographic Module. See FIPS 140-3 Images (#4997).Message may not contain newlines, the signing certificate URL must be a regional SNS host with a .pem path, outbound requests have a timeout and a 1 MiB response cap, and messages signed more than an hour ago are rejected (#4926, #5279).dag failed: prefix, and internal errors no longer leak into task run errors (#5178, #5302, #5212).skip_if and cancel_if conditions are evaluated when a parent returns empty output, empty parent outputs still appear in the parents map, and an invalid CEL condition is ignored instead of failing the DAG (#5295, #5296).CANCEL_* concurrency strategies no longer cancel a task's own retry (#5229).REQUEUED_RATE_LIMIT events are reported at most once a minute per task (#5144, #5138, #5245, #5170).wait_for condition without a base returns an error instead of crashing the handler (#5201, #5260).is_dag_orchestrator is recorded on task events (#5161, #5176, #5199).null payloads are no longer written to the payloads table (#5252, #5253).WithToken overrides the API token for the OTel collector (#5234, #5217).REQUEUED_RATE_LIMIT monitoring events to once per task per minute by @mnafees in #5170-fips images built against the validated Go crypto module by @mnafees in #4997is_dag_orchestrator col on the task events by @mrkaye97 in #5199GetTaskDurationsByTaskIds could get wrong row for taskId by @juliusgeo in #5227timeout_func_only = true for pytest by @abelanger5 in #5247null::jsonb payloads by @mrkaye97 in #5253Note truncated.
Nothing published for this version
refactor(engine): run trigger preflight before the owned transaction …
refactor(engine): run trigger preflight before the owned transaction …
Nothing published for this version
initial commit forgot files generate some more fixes
initial commit
forgot files
generate
some more fixes
perf(ticker): stop rewriting already-owned cron and scheduled rows on…
perf(ticker): stop rewriting already-owned cron and scheduled rows on…
fix: send errors verbatim
fix: send errors verbatim (#5302)
fix(engine): [dag operator] return outputs even when empty, and ignor…
fix(engine): [dag operator] return outputs even when empty, and ignor…
fix: return empty outputs
fix: return empty outputs
Nothing published for this version
feat: add index on cron name directly
feat: add index on cron name directly
fix: down
fix(engine): reject negative worker slot units
fix(engine): reject negative worker slot units (#5274)
fix(engine): handle durable wait_for conditions without base
fix(engine): handle durable wait_for conditions without base (#5260)
Nothing published for this version
feat(engine): gRPC operator service for out-of-process operators
Adds v1.OperatorService so an operator can run outside the engine and talk to
it over gRPC with a tenant-scoped token:
Repository: CreateWorkerOpts.OperatorId flows into the existing CreateWorker
query and skips WORKER / WORKER_SLOT metering, since operator workers are
excluded from those limit counts. AddWorkerActions / RemoveWorkerActions apply
deltas in one transaction under the worker row lock and maintain the action
hash as the XOR of per-action sha256 digests, so equal sets hash equal in any
order and a delta costs O(chunk).
The proto keeps importing the package-less dispatcher.proto for the shared
action types (no SDK regeneration). SERVER_GRPC_OPERATORS_ENABLED (default
false) gates service registration.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
Go client for v1.OperatorService in pkg/client, on the existing reconnecting
stream machinery: Connect calls Register and opens Listen with the start
message; heartbeats and action deltas go through retrySend so a dead stream
reconnects and resumes the worker.
AddActions / RemoveActions never block: a flusher coalesces pending ops (an
add followed by a remove of an id that was never sent cancels out), chunks
them to 1000 ids and sends every 250ms or as soon as a chunk is full. Flush
waits for the queue to drain. The session keeps the desired action set and
replays it as chunked deltas when a reconnect's Register did not resume the
previous worker. PutWorkflow is a thin helper over the admin service that
returns the derived action ids for the caller to AddActions.
Harness-based e2e suite covering echo, durable memo, deactivation on close,
bulk deltas with hash consistency, resume without replay, non-resume replay,
and the operator id and worker ownership checks.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
Adds the v1_serverless_endpoint, v1_serverless_tenant,
v1_serverless_process and v1_serverless_lease tables with a SERVERLESS
operator kind, the ServerlessRepository (also constructible from a
standalone pgxpool like the OLAP repository), data-only encryption
constructors, and a shared pgx pool builder for out-of-process binaries.
Leases are per (tenant, shard) unit and defer owner liveness to a per
process heartbeat row, so holding a unit costs no writes; claims use
FOR UPDATE SKIP LOCKED over the unowned and dead-process sets only.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
Tenant-scoped REST endpoints to create, list, get, update and delete
serverless endpoints, plus per-tenant serverless settings (shard count).
Signing secrets are encrypted at rest and never returned; each endpoint
receives a unique namespace at creation.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
Adds pkg/serverlessoperator: (tenant, shard) leasing with a per-process
heartbeat row, per-unit registrations against a Link interface, a
namespace-prefixed routing cache per tenant, signed healthcheck polling
that registers workflows with the engine, signed POST delivery with
status mapping, and the hatchet-serverless-operator binary. The gRPC
link maps tenants to tokens through a TenantTokenExchange (YAML-backed
LocalExchange in v1). Durable delivery and the in-engine link follow.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
A durable invocation is one request: the operator dials trigger_url with
a signed websocket upgrade, sends the action as the first frame, relays
DurableTaskRequest and DurableTaskResponse frames between the endpoint
and the engine, and reads the endpoint's final done frame as the
terminal outcome. The gRPC link backs the per-invocation channel with
the client's DurableTaskListener. The websocket dial goes through the
safeclient policy dialer so SSRF hardening applies to upgrades.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
Adds the engine link for the serverless operator core: registrations are
operator-backed dispatcher sessions, workflows are put through the admin
service, events go straight to the dispatcher, and durable channels use
the in-process durable task registration. Enabled with
SERVER_SERVERLESS_OPERATOR_ENABLED (default false) and configured with
SERVER_SERVERLESS_OPERATOR_* settings; the core runs with the dispatcher
id as its process id and stops before the dispatcher drains.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
Three defects found by the end-to-end suite:
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
Harness-based e2e scenarios run against both links (in-engine and gRPC):
echo routing by namespace, same-named workflows in one tenant, durable
memo with eviction and replay, durable crash retry, healthcheck-driven
workflow registration, endpoint deletion, lease failover after a crash,
shedding on scale-out, and graceful shutdown. Adds a reference Cloudflare
Worker implementing the endpoint contract for the manual smoke test.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The gRPC operator branch replaced full action set updates with streamed
add/remove deltas, dropped PutWorkflow and UpdateWorker from
OperatorService, and fenced listener sessions with a session id. The
serverless link seam follows:
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
Per-endpoint concurrency is going to be governed by namespace-scoped
tokens with their own rate limits, so the endpoint rows no longer carry
slots and durable_slots and the operator no longer holds a semaphore per
endpoint around delivery. Worker slot config stays a process-level
setting on the registration.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
api-contracts/v1/serverless.proto is the single source for every message the
serverless operator exchanges with an endpoint: the healthcheck request and
response, the non-durable trigger request and its error override, and the
durable websocket frame (a oneof of first, request, response, error and done).
Messages travel as protojson, so the Go operator and the TypeScript package
read and write the same encoding from the same definitions.
The Go bindings are generated into internal/services/shared/proto/v1 with the
same dual proto path as operator.proto, since both reference .AssignedAction
from the package-less dispatcher.proto. The TypeScript bindings live in
sdks/typescript-serverless, the future @hatchet-dev/serverless package, which
for now holds only the generation script (ts-proto pinned to the SDK's version
and options) and its output under src/generated/proto. The SDK's own protoc
script skips serverless.proto the way it skips operator.proto.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The operator encodes and decodes every endpoint message with protojson through
the generated v1.Serverless* types; pkg/serverlessoperator/contract keeps the
header names, the upgrade signature payload, wire constants and the protojson
helpers, and pkg/serverlessoperator/durable keeps the relay and its close codes.
On the wire this means lowerCamelCase field names (endpointId, invocationCount,
inlineWaitBudgetMs, sdkVersion), the first durable frame wrapped as {"first":
...}, and the done frame's output carried as a JSON string so any JSON value
reaches the engine byte for byte. A done frame whose output is not JSON is a
protocol violation and closes the socket with 4005.
The Cloudflare Workers example mirrors the new shapes by hand, as it does for
the rest of the contract until the package replaces it.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
generate-all runs generate-serverless-proto, which installs the package's
pinned toolchain and regenerates sdks/typescript-serverless/src/generated/proto,
so the generate job's diff check fails when the committed bindings are stale.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The shutdown drain ranges the session map and did a blocking send on each
stream session's fin channel. A Listen handler that exited concurrently
released its session and stopped selecting on fin, so a pointer the drain
had already captured blocked it forever. Operator stream sessions now carry
a done channel that Release closes, and the drain selects on fin or done
through requestFin, so a released session is skipped.
Sends on one stream are serialised by the per-worker send lock, but the
lock was released by the caller's deferred Release as soon as its context
ended, while the SendMsg goroutine was still blocked by flow control. The
next caller then started a second SendMsg on the same stream, which gRPC
forbids. The lock is now released by the send goroutine once SendMsg exits;
a caller whose context ends returns without the lock, and the next caller
fails fast with errFlowControlActive after the lock timeout instead of
overlapping.
AddOperatorStreamSession returns an OperatorStreamSession handle with Fin,
Send and Release. Send shares the send serialisation with the action
fan-out so an operator service can write its own protocol messages on the
stream, and the optional wrap function converts each assigned action into
the stream's server message type before it is encoded.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The Listen protocol had no delta acknowledgement. A delta the stream had
accepted but the engine had not committed (stream ended before the apply,
database error, rejected chunk) was dropped by the client as soon as the
send returned, and a resumed worker never learned about it: the client only
replayed its desired set for a fresh worker, and repeating the same
AddActions was a no-op against the desired set.
Listen now returns a stream of OperatorListenResponse, a oneof of the
dispatcher's AssignedAction and an OperatorActionsAck. Each delta carries a
positive, strictly increasing sequence; the engine applies deltas in stream
order and acks a sequence once the change is committed, and an ack for N
confirms every lower sequence. A delta that cannot be applied, or whose ack
cannot be written, ends the stream instead of leaving it unconfirmed. The
dispatcher session wraps actions into the envelope and writes acks through
the same per-stream send serialisation as the fan-out.
The client keeps every sent chunk until its ack arrives and replays the
unacked chunks in order on every reconnect. When the reconnect did not
resume the worker it replaces the queue with a snapshot of the desired set,
taken under the same lock AddActions and RemoveActions use, so a removal
that races the replay is queued relative to the snapshot instead of being
coalesced away with an add that was never sent. Replay runs as the
reconnecting stream's replay callback, under sendMu before the stream is
published, so no chunk is sent in between. A chunk that waits longer than
the ack timeout hangs the stream up so the reconnect replays it. Flush now
waits for the acks, so a nil Flush means the engine committed the set.
Because Flush is called before Actions by the serverless link, the session
runs its receive and heartbeat loops from connect until Close; the receive
loop demultiplexes acks and actions, buffering actions in an inbox that the
consumer Actions attaches drains. Starting the loops and attaching the
consumer add to the WaitGroup under the mutex Close takes, which removes the
Actions/Close race on the WaitGroup; Actions after Close is refused.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The worker action hash was the XOR of per-action digests seeded from
whatever the row held. A worker created with an initial set stored the
sha256 of the ordered list while a worker built by deltas stored the XOR
fold, so equal sets hashed differently across the two paths, and the XOR
fold is linear, so a chosen set of ids could be solved to collide with any
target hash. GetWorkerActionsByWorkerActionHash shares one worker's action
list with every worker of the same hash, so both properties matter.
hashActions is now the canonical digest: sha256 over the lower-cased,
deduplicated ids sorted by byte order, each followed by ";". The delta
transactions recompute the same digest in SQL from the linked rows
(ComputeWorkerActionHash, sha256 over string_agg ordered with the C
collation) while holding the worker's row lock, so the stored hash is
always the digest of the links the transaction leaves behind. UpdateWorker,
which only adds links, recomputes it the same way. At 100,000 linked
actions one add chunk of 1,000 costs about 85 ms, a no-op re-add about
12 ms and a remove about 65 ms on the review database.
Deltas no longer upsert the shared Action rows: existing rows are read, the
missing ones are inserted in sorted order with ON CONFLICT DO NOTHING, and
the resulting row ids are linked in sorted order. Concurrent replays of the
same set by different workers therefore take no locks on existing Action
rows, write no new row versions, and cannot deadlock.
The row lock, the link and the unlink statements all carry the tenant:
LockWorkerActionHash finds no row for a worker of another tenant, which is
reported as an error wrapping pgx.ErrNoRows before anything is mutated, and
the link and unlink statements join Worker and Action on the tenant so a
cross-tenant pair affects zero rows.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The durable stream wrapper checked worker ownership on the first
register_worker only and passed every later message through, so a second
register_worker naming another operator's worker in the same tenant reached
the dispatcher, which rebinds the named worker's durable dispatcher without
an operator check.
The SDK's durable listener registers exactly once per stream and opens a
new stream to register again, so a second register_worker is a protocol
error: the wrapper now refuses it with InvalidArgument, whichever worker it
names, and the stream stays bound to the worker it was authorized for.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
Public gRPC admission had no cumulative limit: an authenticated tenant
token could open any number of Listen streams for one operator, each a
live worker exempt from the worker and slot metering, and stream deltas
that grow the operator's action links without bound.
The service now enforces two per-operator caps, refused with
ResourceExhausted. WithMaxListenStreamsPerOperator bounds the Listen
streams one operator holds open on this replica (default 100); a stream
over the cap is refused before its worker is activated and the slot is
returned when the stream ends. WithMaxActionsPerOperator bounds the action
links held across all workers of the operator (default 1,000,000). The
budget is read once per stream with CountOperatorWorkerActions and then
tracked from the deltas the stream applies; the repository enforces it on
the links a delta would newly create, inside the delta's transaction, so a
delta over the budget links nothing and a delta that repeats actions the
worker already holds (a replay after a resumed reconnect) never counts.
Zero disables either cap.
The caps are options on the service. Wiring them to server configuration
(runtime.grpcOperatorMaxListenStreamsPerOperator and
runtime.grpcOperatorMaxActionsPerOperator with env bindings, next to
grpcOperatorsEnabled) is a change to pkg/config/server and
cmd/hatchet-engine outside this branch's scope.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The relay recorded eviction acks and engine errors as independent flags
that only the closed-without-done path consulted. A request timeout, missed
pongs, backpressure, a link failure or a protocol violation after an
acknowledged eviction turned into a retryable FAILED event for an
invocation the engine had already evicted, and a done frame with output
after an engine error revived a permanently failed invocation
(correctness F10). A done {"status": "evicted"} sent without any eviction
ack was accepted as an eviction, so the terminal event was suppressed for a
task the engine still considered running (correctness F11).
The relay now tracks a phase (running, eviction acked, engine errored) that
moves forward once, on the pump goroutine, before the frame announcing it
is queued, and every way the relay can end is an exit value. finish
resolves (phase, exit) through one table in outcome.go:
Close codes stay the exit's own. The engine-link failure detail moves to
the operator log; the task error and the close reason say "engine link
failed". TestResolveTable covers every (phase, exit) pair and the two
reproductions from the audit are regression tests with a goleak check on
the relay lifecycle.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
gorilla parses the upgrade response with http.ReadResponse on the raw
connection, outside any net/http header budget, and the relay's frame
limit only applies once the websocket exists. An endpoint could therefore
make the operator allocate an arbitrary amount of memory per handshake
(security F01).
The dialer now installs a headerLimitConn under both NetDialContext (ws,
development only) and NetDialTLSContext (wss). For wss the TLS session is
negotiated here with the URL's host name, so the wrapper sits on the
decrypted stream and budgets exactly the bytes the handshake parser sees,
up to and including the blank line that ends the response head; frames
after that are not counted. Crossing the budget fails the dial with
ErrUpgradeHeadersTooLarge, which the relay reports as a non-retryable
failure naming the limit.
The limit is Params.MaxUpgradeHeaderBytes, 64 KiB by default, exposed as
Config.WSMaxUpgradeHeaderBytes, SERVERLESS_OPERATOR_WS_MAX_UPGRADE_HEADER_BYTES
on the binary and SERVER_SERVERLESS_OPERATOR_WS_MAX_UPGRADE_HEADER_BYTES on
the engine. The regression test answers the upgrade with a 16 MiB header
over plain and TLS sockets.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The send queue held up to 256 encoded frames with no bound on their size,
so a slow endpoint with large engine responses could pin hundreds of
megabytes per relay (performance F11). The relay now charges every frame
against Params.MaxQueuedBytes before queueing it and releases the charge
once the writer has taken it off the queue; crossing the budget ends the
relay with the existing CloseBackpressure code, the frame count limit
stays as it was.
The budget is 16 MiB by default and configurable as
Config.WSMaxQueuedBytes, SERVERLESS_OPERATOR_WS_MAX_QUEUED_BYTES on the
binary and SERVER_SERVERLESS_OPERATOR_WS_MAX_QUEUED_BYTES on the engine.
The regression test parks the writer and queues eight 16 KiB frames
against a 64 KiB budget.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
Three findings share this package.
Idle pool (correctness F19, performance F7): both transports were zero
literals, so MaxIdleConns and IdleConnTimeout were unbounded and one
operator process talking to an ever-growing set of origins kept a socket
per origin until the far side closed it. The transport now takes
Config.MaxIdleConns (256), MaxIdleConnsPerHost (4) and IdleConnTimeout
(90s), and Sender.CloseIdleConnections releases the pool; the binary binds
SERVERLESS_OPERATOR_HTTP_MAX_IDLE_CONNS, _PER_HOST and
_IDLE_CONN_TIMEOUT and closes the pool on exit. The engine's server config
carries the same three settings. safeurl clones the transport it is given,
so the Sender keeps the clone for CloseIdleConnections.
Response cap (correctness F20, security F11): an oversize response was
drained to EOF before ErrResponseTooLarge was returned, so a streaming
endpoint held the delivery slot until the caller's deadline. Deliver now
reads one byte past the cap and returns; the deferred body close drops
the connection instead of draining it. An 8-byte cap consumes at most 9
bytes and a stalled 1025-byte stream against a 1024-byte cap returns
immediately.
Redaction (security F12): dial, DNS and read failures were returned with
the resolved address, port, path and socket error, and those strings end
up in task errors and endpoint status. Transport failures are now
returned as *EndpointError, which names the configured host and the
stage (resolve, connect, tls, read, request) and, for certificate
verification failures, the x509 message the tenant needs; the full error
goes to the operator log. errors.Is and errors.As still reach the cause.
A blocked resolution no longer names the address it resolved to. The
durable dialer applies PublicError to its own TLS handshake errors.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The relay stamped the task id and invocation on every nested request but
forwarded the workflow names in trigger_runs and the user event keys in
wait_for conditions as the endpoint sent them, so a durable task could
trigger any workflow and wait for any event of the tenant (security F13).
stamp now ends with confine, in the shared relay so both links get it:
trigger option names and user event keys are prefixed with the endpoint's
namespace through contract.ApplyNamespace, the same rule the operator
applies to what it registers (a name that already carries the prefix is
left alone). Engine-generated identifiers are untouched: log entry refs,
readable data keys and or-group ids label the task's own log, sleep
conditions name no resource, event scopes are matched within the
namespaced key, and memo keys are private to the task. The relay comment
documents this as the confinement boundary.
contract.NamespaceSeparator, NamespacePrefix and ApplyNamespace carry the
rule; routing.go's prefixName implements the same one and should call
ApplyNamespace once that package is touched.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The reference Worker verified the HMAC on POST bodies but never looked at
the timestamp the signature covers, accepted upgrade timestamps from the
future, kept no nonce set, and ran whatever task the first frame named
regardless of the task the upgrade was signed for (security F07).
The contract now states the freshness rule in contract/http.go: the
timestamp inside every signed POST body and the upgrade's
X-Hatchet-Timestamp must lie within RequestMaxAge (5 minutes) of the
endpoint's clock in either direction; UpgradeMaxAge is the same window.
Endpoints consume the upgrade nonce after the signature verified and check
the first frame's task id and invocation against the verified headers.
The Worker example implements it: verifySignedBody (HMAC, then the body's
timestamp, then the endpoint id when HATCHET_ENDPOINT_ID is set) replaces
the bare HMAC check in both POST handlers; verifyUpgradeSignature rejects
future timestamps and consumes the nonce from a bounded, expiring NonceSet
(4096 entries per isolate, documented as needing a Durable Object or KV
across isolates); DurableClient.assertMatches closes the socket with 1008
when the first frame does not name the verified task and invocation. The
README documents the checks and the idempotency key for side effects.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The claim statement now decides which units are claimable in its own
snapshot: units with no owner, plus units whose owner's heartbeat row has
expired, and it requires the claimer's own row to be live. A dead-process
list read earlier is no longer a claim input, so an owner that heartbeats
between another process's look and its claim keeps its units (correctness
F04), and an expired or swept process cannot claim until it heartbeats.
Deleting a process row, by the periodic sweep or a graceful shutdown,
releases whatever the row still owns in the same statement, and the
migration releases units whose owner row is already gone, so a swept
owner never strands units (correctness F03). Count and claim share one
liveness rule.
The claim no longer sorts the candidate population: unowned units are
walked in key order from a random start key with a wraparound, through a
covering partial index, and units of expired processes are walked per
dead process through the owner index (performance F2). On the perf
session's fixtures a claim of 64 from one million unowned units went from
551 ms with a 47 MB external sort to 2 ms, and a claim among 100k
dead-owned units from 115 ms to 2 ms. The per-tick claim budget is the
process's share of the claimable units (claimable divided by live
processes) bounded by MaxClaimPerTick, taken in statements of at most
ClaimBatch units, instead of a fixed 128; the claimable count is windowed
at the fleet's budget for one tick.
The leaser retries its initial heartbeat with backoff instead of exiting
(groundwork for correctness F16), and a heartbeat that lands more than
the TTL after the previous one triggers an immediate reconcile so lost
units are noticed without waiting for the next tick.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The tenant action union is now a reference count per action (how many
enabled endpoints advertise it) with a revision and a bounded delta log.
An action enters the union on the 0 to 1 transition and leaves it on
1 to 0, so a healthcheck change costs that endpoint's actions and every
batch of rows (a load, a refresh, a page of a gained unit) publishes at
most one revision; the sorted form is built once per revision on demand.
Registrations advertise a revision and catch up from the delta log,
falling back to a full diff only when the log no longer reaches back, so
an unchanged union costs a sync a revision comparison (performance F1,
F5). Loading 3,000 endpoints with 10 actions each went from 19.5 s and
18.3 GB to under the 200 ms and 50 MB budget the regression test holds
it to; 100 unchanged syncs of a 100k-action union went from 860 MB of
allocation to none.
The routing refresh is keyed on the row version, the later of updated_at
and status_changed_at, with a (version, id) keyset and a matching
expression index instead of a one-second overlap window, so an unchanged
tenant reads nothing (performance F4: 609 ms and a 88 MB sort for a
100k-row tenant to 0.06 ms) and status transitions written by another
owner surface within a refresh. Status writes return the database's
status_changed_at and the cache keeps a local write only while it is
newer than the row's, so a takeover sees the previous owner's status and
recovers the endpoint (correctness F18).
A poll now computes one outcome after both the healthcheck and the
catalog application and writes the status once when it differs from the
row; an unchanged catalog the engine rejected is retried with backoff
instead of re-put on every poll, so a persistently rejected endpoint is
one write, not a healthy/unhealthy pair per poll (performance F12). A
refused action delta restores the endpoint's previous contribution so a
rejected catalog never stays in the shared union, and the per-poller
workflow hash memory is pruned to the accepted catalog.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The runner now keeps one registration per tenant per process, shared by
every unit of the tenant the process owns (performance F6); it opens with
the first unit, stays while any unit remains and closes with the last.
The links open registrations by tenant alone, so the in-engine worker is
named per dispatcher and the shard label is gone.
The runner's lock covers its maps only. Each tenant's work (gaining and
losing units, maintenance, shutdown) runs under the tenant's own
operation lock, so a hung engine on one tenant's open, a slow poller
shutdown or a database outage no longer stalls other tenants or the
lease tick, and maintenance refreshes tenants with bounded parallelism
(performance F9).
A gained unit whose tenant fails to load is kept pending and retried by
every maintenance pass until it is served, so a claimed unit is never
left owned but unserved (correctness F02). Healthchecks take a per-tenant
slot inside the process-wide limit, first polls of a gained unit are
spread out, and the network slots are released before a catalog is
applied under its own deadline (performance F8, security F05).
In-flight deliveries are keyed by task and attempt (retry count and
durable invocation). A finish removes only its own record, a cancel
reaches the attempt it names (or every attempt of the task when it names
none in flight), and a duplicate assignment of an attempt in flight is
ignored (correctness F09). The endpoint's request timeout is applied
before the durable channel is opened, so the in-engine handshake is
bounded like the rest of the invocation (correctness F15).
Every advertised action, derived from a workflow or listed, is validated
with the engine's action rules before the namespace prefix, and a
healthcheck may advertise at most MaxWorkflowsPerEndpoint workflows and
MaxActionsPerEndpoint actions; a catalog over either cap is refused and
the endpoint marked with the error (security F04, F05).
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
Durable requests reach the shared listener through a bounded per-hub
queue and one pump goroutine, since the listener's own enqueue cannot be
cancelled; every channel's Send selects on the invocation's close signal
instead, so a full queue (an unreachable engine) no longer holds a
closing invocation, its relay teardown or the operator's shutdown
(correctness F12). A send refused by the close undoes its pending ack and
in-flight slot.
Every goroutine a channel starts (ack waits, entry waits, eviction acks)
is refused once closing begins and joined before the invocation's pending
state is dropped from the listener, and an entry wait that finds the
channel closed drops its registration again, so no callback can outlive
its invocation on the shared listener (correctness F13). A goleak test
covers the ordinary lifecycle.
A tenant's cached client is closed when it is replaced by a rotated token
or released with the tenant's last registration, through the client's
Close when it exposes one (correctness F14, performance F13); pkg/client
does not expose its gRPC connection yet, which the regression test
reports by skipping.
Open refuses a registration the engine authenticated as another tenant
than the unit's, closing the session before any action is sent, and
refuses a token whose tenant claim names another tenant before a client
is built (security F09). The local token exchange records the mtime of
every referenced token_file and reloads when any of them changes, and
logs reload failures when given a logger (security F10).
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The dispatcher registers a durable task for routing before the
register-worker handshake completes, so a completion restored for a
resumed invocation can reach the channel before the ack. openDurable now
holds such responses for the invocation under the existing ack-before-
entry ordering, bounded by handshakeHoldLimit, instead of rejecting them
as an invalid handshake (correctness F21, reproduced at the link level
with the dispatcher's ordering).
Open creates the worker without actions and links the initial set
through the same chunked bulk path a streamed delta takes, before the
worker is activated, instead of one action upsert per action inside
worker creation (performance F3); both links now build the action set
from the same empty seed.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The engine's serverless goroutine ran the core once: an initial failure
(a database unreachable at startup) ended it for the life of the engine,
and its stop function could return that stale error ahead of the rest of
the cleanup chain (correctness F16). The core is now supervised: a stop
the engine did not ask for is logged and the core restarted with backoff
(one second doubling to thirty, reset after a long run), the supervisor
exposes whether the core is running for a readiness probe, and stop ends
the core, waits for it and never reports earlier failures, so the
dispatcher cleanup continues past it. The leaser's initial heartbeat
retries with backoff on its own as well.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
link_test.go carries no build constraint while the regression file is
excluded under the integration and e2e tags, so the helper both use
lives in link_test.go; go vet -tags integration compiles again.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
Client gains Close, which closes the gRPC connection the client owns, so
a caller that retires a client (the serverless grpclink when a tenant's
token rotates or its last registration closes) releases the connection's
goroutines instead of leaving them behind (correctness F14, performance
F13). grpclink's closeClient calls it directly and the goleak regression
test that had to skip while pkg/client exposed no Close now runs.
DurableTaskListener.SendRequest takes a context and returns an error:
the request queue is bounded, so a caller blocked on an unreachable
engine is released when its context ends or the listener is stopped
(Stop closes a per-generation stop signal that Start renews), and the
listener's own request helpers drop their pending ack when the send is
refused. SendMemoCompleted follows the same shape. grpclink's pump hands
requests to the listener under a context closeAll cancels, so a stopped
registration no longer holds the pump on a full queue (correctness F12).
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The hub indexed open channels by (task, invocation) in one map, so a
server-evict notice, which names one task, scanned every open channel of
the registration under the hub's mutex. Channels are now indexed by task
and then invocation: a notice visits the named task's entries only, a
task is dropped from the index when its last channel closes, and closeAll
walks the nested index. A test opens channels for two tasks and checks
that only the named task's invocations at or below the notice receive
the server_evict response.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The relay's upgrade header budget and queued bytes budget were
configurable on the core, the binary and the engine, but the
registration's durable.Run call never passed them, so the relay always
used its built-in defaults. Registration now passes
Config.WSMaxUpgradeHeaderBytes and WSMaxQueuedBytes as
Params.MaxUpgradeHeaderBytes and MaxQueuedBytes, and the engine link's
ConfigFromServer copies both from the server config, so
SERVERLESS_OPERATOR_WS_MAX_UPGRADE_HEADER_BYTES and WS_MAX_QUEUED_BYTES
(and their SERVER_SERVERLESS_OPERATOR prefixed forms) are live.
The in-engine mode's safeclient sender now takes the server config's
HTTPMaxIdleConns, HTTPMaxIdleConnsPerHost and HTTPIdleConnTimeout, as the
out-of-process binary already did, and the stop function closes the idle
pool once the core has drained (correctness F19, performance F7).
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
routing.go's prefixName and namespacePrefix carried their own copy of
the namespace rule the durable relay applies to nested references.
Both now delegate to contract.ApplyNamespace and contract.NamespacePrefix,
and ParseNamespace reads contract.NamespaceSeparator, so what the
operator registers and what the relay confines an endpoint to are
prefixed by one implementation (security F13 follow-up).
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The per-operator Listen stream cap and action link budget were options
on the operator service with no way to set them from configuration.
Runtime now carries GRPCOperatorMaxListenStreamsPerOperator (default
100) and GRPCOperatorMaxActionsPerOperator (default 1,000,000), bound to
SERVER_GRPC_OPERATOR_MAX_LISTEN_STREAMS_PER_OPERATOR and
SERVER_GRPC_OPERATOR_MAX_ACTIONS_PER_OPERATOR next to
SERVER_GRPC_OPERATORS_ENABLED, and both engine run paths pass them to
grpcoperator.New through WithMaxListenStreamsPerOperator and
WithMaxActionsPerOperator. Zero disables either cap (security F06).
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
UpdateOperatorWorkerActions stored the digest of the list it was given
while only adding links, so a worker that already held links ended with a
hash that named a subset of its linked set, and
GetWorkerActionsByWorkerActionHash would then share the wrong action list
with other workers of that hash. The call now locks the worker row with
the tenant in the predicate (a worker of another tenant is refused and
nothing is written), lower-cases and dedupes the ids, links through the
tenant-checked LinkActionsToWorkerReturning, and recomputes the hash from
the linked rows with ComputeWorkerActionHash inside the same transaction,
as the worker repository's delta path does (correctness F17, security
F02, F08). Tests check that the stored hash is the canonical digest of
the final linked set after an additive call and after a call naming a
subset, that it equals the delta path's hash for the same set, and that
the tenant is checked.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
CountOperatorWorkerActions joins the action links of every worker of an
operator once per Listen stream, finding the workers by (tenantId,
operatorId); "Worker" had no index on that pair. v1_0_156 adds
Worker_tenantId_operatorId_idx (a plain CREATE INDEX inside the
migration's transaction, the migration is unreleased) and the schema
mirrors it.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The supervised in-engine core exposed Running for a readiness probe but
nothing consulted it. health.Health now accepts named readiness checks
(AddReadinessCheck), which /ready evaluates after the queue and the
repository and logs by name when one fails; /live is unchanged.
startServerlessOperator returns the supervisor's Running alongside stop,
and both engine run paths register it as the "serverless-operator" check
when the health server is on, so a replica whose core is down and waiting
to restart reports not ready until the core is back (correctness F16).
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
Four settings the fix branches added to the engine's
ServerlessOperatorConfigFile had no environment binding
(HEALTHCHECK_TENANT_CONCURRENCY, HEALTHCHECK_APPLY_TIMEOUT,
MAX_WORKFLOWS_PER_ENDPOINT, MAX_ACTIONS_PER_ENDPOINT); they are bound
under SERVER_SERVERLESS_OPERATOR_ now.
MaintenanceConcurrency existed on the core Config only. It is now a
setting of the binary (SERVERLESS_OPERATOR_MAINTENANCE_CONCURRENCY) and
the engine (SERVER_SERVERLESS_OPERATOR_MAINTENANCE_CONCURRENCY), default
8. The leaser's MaxClaimPerTick existed on lease.Config only; the core
Config carries it as LeaseMaxClaimPerTick (default 1024, the leaser's
own default) and passes it to the leaser, and both binaries bind it as
LEASE_MAX_CLAIM_PER_TICK. ConfigFromServer copies both, and its test now
sets and checks every shared setting so a field added to one side
without the other fails.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
fakeOperatorStore counted GetOperatorById calls and served its map with
no synchronisation, while TestListenStreamCapIsPerOperator authorizes
several Listen streams concurrently through it, so the race detector
failed the package intermittently. The counter is atomic and the map is
guarded, and the assertions read the counter through Load.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The test rewrote the tenant token file with a truncate-then-write while
the exchange polled its mtime every ten milliseconds, so the poller could
reload between the two and parse an empty file as a valid mapping with no
tenants, which failed the later lookup intermittently. The helper now
writes a temporary file and renames it over the path, the way a rotated
secret or a mounted file is swapped, which is what the exchange is
documented to follow.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The shutdown drain ranges the session map and did a blocking send on each
stream session's fin channel. A Listen handler that exited concurrently
released its session and stopped selecting on fin, so a pointer the drain
had already captured blocked it forever. Operator stream sessions now carry
a done channel that Release closes, and the drain selects on fin or done
through requestFin, so a released session is skipped.
Sends on one stream are serialised by the per-worker send lock, but the
lock was released by the caller's deferred Release as soon as its context
ended, while the SendMsg goroutine was still blocked by flow control. The
next caller then started a second SendMsg on the same stream, which gRPC
forbids. The lock is now released by the send goroutine once SendMsg exits;
a caller whose context ends returns without the lock, and the next caller
fails fast with errFlowControlActive after the lock timeout instead of
overlapping.
AddOperatorStreamSession returns an OperatorStreamSession handle with Fin,
Send and Release. Send shares the send serialisation with the action
fan-out so an operator service can write its own protocol messages on the
stream, and the optional wrap function converts each assigned action into
the stream's server message type before it is encoded.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The Listen protocol had no delta acknowledgement. A delta the stream had
accepted but the engine had not committed (stream ended before the apply,
database error, rejected chunk) was dropped by the client as soon as the
send returned, and a resumed worker never learned about it: the client only
replayed its desired set for a fresh worker, and repeating the same
AddActions was a no-op against the desired set.
Listen now returns a stream of OperatorListenResponse, a oneof of the
dispatcher's AssignedAction and an OperatorActionsAck. Each delta carries a
positive, strictly increasing sequence; the engine applies deltas in stream
order and acks a sequence once the change is committed, and an ack for N
confirms every lower sequence. A delta that cannot be applied, or whose ack
cannot be written, ends the stream instead of leaving it unconfirmed. The
dispatcher session wraps actions into the envelope and writes acks through
the same per-stream send serialisation as the fan-out.
The client keeps every sent chunk until its ack arrives and replays the
unacked chunks in order on every reconnect. When the reconnect did not
resume the worker it replaces the queue with a snapshot of the desired set,
taken under the same lock AddActions and RemoveActions use, so a removal
that races the replay is queued relative to the snapshot instead of being
coalesced away with an add that was never sent. Replay runs as the
reconnecting stream's replay callback, under sendMu before the stream is
published, so no chunk is sent in between. A chunk that waits longer than
the ack timeout hangs the stream up so the reconnect replays it. Flush now
waits for the acks, so a nil Flush means the engine committed the set.
Because Flush is called before Actions by the serverless link, the session
runs its receive and heartbeat loops from connect until Close; the receive
loop demultiplexes acks and actions, buffering actions in an inbox that the
consumer Actions attaches drains. Starting the loops and attaching the
consumer add to the WaitGroup under the mutex Close takes, which removes the
Actions/Close race on the WaitGroup; Actions after Close is refused.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The worker action hash was the XOR of per-action digests seeded from
whatever the row held. A worker created with an initial set stored the
sha256 of the ordered list while a worker built by deltas stored the XOR
fold, so equal sets hashed differently across the two paths, and the XOR
fold is linear, so a chosen set of ids could be solved to collide with any
target hash. GetWorkerActionsByWorkerActionHash shares one worker's action
list with every worker of the same hash, so both properties matter.
hashActions is now the canonical digest: sha256 over the lower-cased,
deduplicated ids sorted by byte order, each followed by ";". The delta
transactions recompute the same digest in SQL from the linked rows
(ComputeWorkerActionHash, sha256 over string_agg ordered with the C
collation) while holding the worker's row lock, so the stored hash is
always the digest of the links the transaction leaves behind. UpdateWorker,
which only adds links, recomputes it the same way. At 100,000 linked
actions one add chunk of 1,000 costs about 85 ms, a no-op re-add about
12 ms and a remove about 65 ms on the review database.
Deltas no longer upsert the shared Action rows: existing rows are read, the
missing ones are inserted in sorted order with ON CONFLICT DO NOTHING, and
the resulting row ids are linked in sorted order. Concurrent replays of the
same set by different workers therefore take no locks on existing Action
rows, write no new row versions, and cannot deadlock.
The row lock, the link and the unlink statements all carry the tenant:
LockWorkerActionHash finds no row for a worker of another tenant, which is
reported as an error wrapping pgx.ErrNoRows before anything is mutated, and
the link and unlink statements join Worker and Action on the tenant so a
cross-tenant pair affects zero rows.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The durable stream wrapper checked worker ownership on the first
register_worker only and passed every later message through, so a second
register_worker naming another operator's worker in the same tenant reached
the dispatcher, which rebinds the named worker's durable dispatcher without
an operator check.
The SDK's durable listener registers exactly once per stream and opens a
new stream to register again, so a second register_worker is a protocol
error: the wrapper now refuses it with InvalidArgument, whichever worker it
names, and the stream stays bound to the worker it was authorized for.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
Public gRPC admission had no cumulative limit: an authenticated tenant
token could open any number of Listen streams for one operator, each a
live worker exempt from the worker and slot metering, and stream deltas
that grow the operator's action links without bound.
The service now enforces two per-operator caps, refused with
ResourceExhausted. WithMaxListenStreamsPerOperator bounds the Listen
streams one operator holds open on this replica (default 100); a stream
over the cap is refused before its worker is activated and the slot is
returned when the stream ends. WithMaxActionsPerOperator bounds the action
links held across all workers of the operator (default 1,000,000). The
budget is read once per stream with CountOperatorWorkerActions and then
tracked from the deltas the stream applies; the repository enforces it on
the links a delta would newly create, inside the delta's transaction, so a
delta over the budget links nothing and a delta that repeats actions the
worker already holds (a replay after a resumed reconnect) never counts.
Zero disables either cap.
The caps are options on the service. Wiring them to server configuration
(runtime.grpcOperatorMaxListenStreamsPerOperator and
runtime.grpcOperatorMaxActionsPerOperator with env bindings, next to
grpcOperatorsEnabled) is a change to pkg/config/server and
cmd/hatchet-engine outside this branch's scope.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
Client gains Close, which closes the gRPC connection the client owns, so
a caller that retires a client (the serverless grpclink when a tenant's
token rotates or its last registration closes) releases the connection's
goroutines instead of leaving them behind (correctness F14, performance
F13). grpclink's closeClient calls it directly and the goleak regression
test that had to skip while pkg/client exposed no Close now runs.
DurableTaskListener.SendRequest takes a context and returns an error:
the request queue is bounded, so a caller blocked on an unreachable
engine is released when its context ends or the listener is stopped
(Stop closes a per-generation stop signal that Start renews), and the
listener's own request helpers drop their pending ack when the send is
refused. SendMemoCompleted follows the same shape. grpclink's pump hands
requests to the listener under a context closeAll cancels, so a stopped
registration no longer holds the pump on a full queue (correctness F12).
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The per-operator Listen stream cap and action link budget were options
on the operator service with no way to set them from configuration.
Runtime now carries GRPCOperatorMaxListenStreamsPerOperator (default
100) and GRPCOperatorMaxActionsPerOperator (default 1,000,000), bound to
SERVER_GRPC_OPERATOR_MAX_LISTEN_STREAMS_PER_OPERATOR and
SERVER_GRPC_OPERATOR_MAX_ACTIONS_PER_OPERATOR next to
SERVER_GRPC_OPERATORS_ENABLED, and both engine run paths pass them to
grpcoperator.New through WithMaxListenStreamsPerOperator and
WithMaxActionsPerOperator. Zero disables either cap (security F06).
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
UpdateOperatorWorkerActions stored the digest of the list it was given
while only adding links, so a worker that already held links ended with a
hash that named a subset of its linked set, and
GetWorkerActionsByWorkerActionHash would then share the wrong action list
with other workers of that hash. The call now locks the worker row with
the tenant in the predicate (a worker of another tenant is refused and
nothing is written), lower-cases and dedupes the ids, links through the
tenant-checked LinkActionsToWorkerReturning, and recomputes the hash from
the linked rows with ComputeWorkerActionHash inside the same transaction,
as the worker repository's delta path does (correctness F17, security
F02, F08). Tests check that the stored hash is the canonical digest of
the final linked set after an additive call and after a call naming a
subset, that it equals the delta path's hash for the same set, and that
the tenant is checked.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
fakeOperatorStore counted GetOperatorById calls and served its map with
no synchronisation, while TestListenStreamCapIsPerOperator authorizes
several Listen streams concurrently through it, so the race detector
failed the package intermittently. The counter is atomic and the map is
guarded, and the assertions read the counter through Load.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
The gRPC operator service owns logic that is not gRPC's: upserting the operator
row, creating or resuming its worker, activating the worker under a listener
fence, applying action set deltas within the per-operator budget, and closing
the session again. The in-process operator host needs all of it, and the
serverless operator's engine link already re-implements it by hand, with the
drift that comes from a copy: no resume, no budget, a separate dispatcher
session key.
internal/services/operatorsvc now holds that logic behind Service, Session and
the two deliveries a session can have. A stream delivery encodes assigned
actions onto a gRPC stream and counts against the per-operator stream cap; a
handler delivery hands them to an operator running in this process and counts
against nothing but the action budget, since it holds no stream. Both get one
session id that is the dispatcher's session key and the worker row's listener
fence, both notify the scheduler through the same throttle, and both close by
pause, drain, deactivate. The gRPC Listen handler closes without the pause: its
operator pauses itself before hanging up, and a stream that simply drops must
leave the worker assignable for the next connection.
internal/services/grpcoperator keeps what is genuinely the protocol: the
metadata that names the operator, the start handshake, the message loop, the
delta acks, and the durable stream it hands to the dispatcher unchanged.
The dispatcher gains AddOperatorSession, the handler-backed twin of
AddOperatorStreamSession, keyed on the session id its caller chose, and
pkg/operator gains ActionHandler, the one method a session needs to deliver to.
The test doubles for the stores and the dispatcher move to
operatorsvc/operatorsvctest so the service's tests and the handlers' tests
drive the same behaviour.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
An operator hosted in the engine drives a durable invocation over the
dispatcher's channel-backed session rather than the DurableTask stream, and the
handshake and ordering that goes with it is not obvious enough to leave to each
host.
Session.OpenDurable performs the register_worker handshake itself, so Recv only
ever returns invocation traffic, and applies the ordering the operator expects:
the engine delivers a completion for an already satisfied entry as soon as the
invocation registers, which on a replay is before the operator has sent the
wait_for that names it, so an entry is held until the ack carrying its ref has
gone out. Responses that arrive during the handshake are held under the same
rule, bounded at 256 so an engine that never acks cannot make the session buffer
without end. As on the stream, one ack-bearing request is in flight at a time,
because the engine keys its pending state by (task, invocation).
This is a port of the serverless operator's engine link, which grew the ordering
rule as a bug fix and will be deleted in favour of this once the in-process host
lands. The gRPC DurableTask handler is unchanged: it still hands the raw stream
to the dispatcher after the ownership check.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_014gXd9VgU8FwoN3DijfprEK
An operator that wants to shut down cleanly needs the scheduler to stop
assigning to its worker before it stops answering, and it needs to know the
pause is committed before it starts draining. A message on the Listen stream
would need an ack of its own and would not work while that stream is being torn
down or reconnected, so pausing is its own unary call.
OperatorService.PauseWorker takes {worker_id, paused}: the same call pauses and
unpauses, which is what a worker that is resumed rather than replaced needs. It
is authorised like SendStepActionEvent, by the operator metadata plus worker
ownership. Register already clears the pause when it resumes a worker, so an
operator that paused to drain and then crashed comes back assignable.
T…
refactor: rm separate payload file
feat: typed payload shapes
refactor: rm separate payload file
refactor: app-side enum for the type, and then map that to the db
refactor: remove some dead code
refactor: rm more dead code
chore: rm lots more dead code
refactor: simplify the V1StepRunData and read error messages out of the task events
feat: add new columns to db
chore: gen
feat: wiring
refactor: simplify a bit
fix: migration version
fix: handle replay bug
chore: rm unused method
fix: cleanup
chore: rm leftover prior changes
fix: replace helpers
fix: rm default
fix: add an env var to gate
fix: simplify by adding a payload version
fix: simplify more
fix: simplify yet more, remove some tests
fix: make filter payloads compatible across versions
chore: shrink diff more
fix: greptile comment
refactor: rm separate payload file
feat: typed payload shapes
refactor: rm separate payload file
refactor: app-side enum for the type, and then map that to the db
refactor: remove some dead code
refactor: rm more dead code
chore: rm lots more dead code
refactor: simplify the V1StepRunData and read error messages out of the task events
feat: add new columns to db
chore: gen
feat: wiring
refactor: simplify a bit
fix: migration version
fix: handle replay bug
chore: rm unused method
fix: cleanup
chore: rm leftover prior changes
fix: replace helpers
fix: rm default
fix: add an env var to gate
fix: simplify by adding a payload version
fix: simplify more
fix: simplify yet more, remove some tests
fix: make filter payloads compatible across versions
chore: shrink diff more
fix: greptile comment
One column per quarter.