NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
Go modules
Go modules, #1316 by repository stars
Latest stable
v0.40.1
Last release 16 days ago
06 Nov 2026
Ships fairly regularly
a new release about every 2 weeks
Nearly every release is documented
notes for 35 of 37 stable releases
Nothing withdrawn
no release was ever pulled
9 months old
61 releases · first in 2026
Drag across the ruler to compare, or tap a release to open it.
fix(execution): disarm first-event watchdog without canceling request by @spboyer in https://github.com/microsoft/waza/pull/884
Full Changelog: https://github.com/microsoft/waza/compare/v0.40.0...v0.40.1
Full Changelog: v0.40.0...v0.40.1
Your coding agent can read these notes before it upgrades. Set up the MCP server →
One column per month.
All notable changes to waza will be documented in this file.
The format is based on Keep a Changelog,
and this project adheres to Semantic Versioning.
inline.copilot-text execution through the owned production SDK engine for one declared shard/task/trial/attempt (#758). Requires explicit model and paid consent; tools and skills are disabled and general native file/tool/skill workflows remain unsupported. Paid HTTP execution additionally requires a loopback peer, localhost/loopback Host and same-origin browser requests; remote mock and offline routes retain their previous behavior. Admitted eval metrics are identity-bound, uncomputed metadata, not aggregate values or threshold decisions. Cancellation and effective execution/shutdown deadlines interrupt the private transport; lifecycle and incomplete-usage diagnostics cannot become success, and isolated sessions never resume with unrestricted defaults solely to retrieve usage. CLI/API, terminal schema and Lab distinguish copilot from mock. Existing mock requests, identity algorithm, local adapters and result formats are unchanged; older servers reject the new capability before execution. SDK-fake conformance is not live-provider or native-platform evidence. See invocation and migration. No release or container publication is implied.Selected AgentV assertion interoperability (v0.40.0 preview) - Add offline, version-pinned waza interop import/export for one explicitly selected case-sensitive contains assertion and its single-check native grader mapping (#789). Retained upstream fixture/oracle controls verify binary decisions and scores; unsupported selected semantics fail explicitly and excluded task/suite semantics are reported. This is not whole-task/suite support: AgentEvals-spec, Harbor, ASSERT and full evidence/provenance round trips remain unsupported, and #789 remains open. See the preview guide.
Local runner container references (v0.40.0 preview) - Keep the default scratch reference target mock/text-only and add an explicit digest-pinned Debian/glibc native-reference target for the exact-unit native Copilot text profile (#762). The real bundled CLI is exercised against an in-container synthetic provider with runtime networking disabled, alongside positive discovery/hook controls, restriction checks, exact selection, timeout, cancellation and process cleanup. Docker CI requires native Linux AMD64; local ARM64 or emulated runs are not equivalent. Neither target is published, and local image IDs are not registry pins. A separate manual verifier is prepared to anonymously pull the published immutable digest, verify source/platform/provenance, and run mock and bundled-native production-caller controls against that exact image; neither workflow has been dispatched, and public-image acceptance remains outstanding. No scheduler or Lab change is introduced. See the quickstart and maintainer update policy.
Bounded local compatibility (v0.40.0 preview) - Add offline runner adapt and opt-in run --runner-compat for one literal mock/text task, trial and attempt (#761). Both use the byte-preserving adapter; local execution uses the shared runner service, current EvaluationOutcome writer schema 1.8 and local 0/1/2 exits. Ordinary run, historical schemas and artifact commands remain available; none are deprecated. Unsupported mappings, flags, ambient configuration, fixtures and adjacent trigger work fail explicitly. Remove --runner-compat to retain native/local behavior, or explicitly choose terminal JSON with runner run. No Lab change is required because no new result evidence/status is introduced. See the compatibility and migration table. This is not a release announcement.
Runner negotiation 1.0 (first v0.40.0 layer) - Add offline JSON stdin/stdout runner capabilities and runner negotiate, matching HTTP APIs, strict request/response schemas, conformance fixtures and live Lab incompatibility diagnostics (#757). Compatibility alone never starts work.
Exact captured mock/text work units (v0.40.0 preview) - Add runner validate / runner run and HTTP equivalents with fail-closed shard/task/trial/attempt admission, one fresh engine call, no task scanning, cache or automatic retries, and live Lab selection validation (#758). Each explicit repeat executes again, not deduplication. Opt-in Copilot/text is described above; general native file/tool/skill workflows remain unavailable. Existing local commands and artifacts are unchanged; no forms are deprecated. See the contract and migration guide.
Input-derived runner identities (v0.40.0 preview) - Admitted validation/execution receipts now carry identity with version sha256-waza-c14n-v1, canonical input digest and run/shard/task/trial/attempt lineage through CLI JSON, HTTP, Lab and validation export (#759). Independent frozen source and producer vectors define the stability boundary. Invalid/unadmitted work has no identity; later execution failures retain it without implying execution success. Historical local results and receipts without identities are unchanged and are not backfilled. The unshipped sha256-json-v1 draft is not compatible; compare version and ID together. See normalization, conformance and migration.
Runner terminal envelope (v0.40.0 preview) - runner run and HTTP /runner/run now emit one schema-validated waza.runner-terminal envelope, with success/evaluation/infrastructure/cancellation/timeout/unsupported/invalid exits 0–6, actual admitted identity and bounded observed verdicts (#760). This replaces only the earlier unshipped run receipt, not offline negotiation/validation or local result artifacts. Lab's explicit mock-run action shows and exports the real response; edits and errors clear stale success. Short/failed output never retries, and SIGKILL/host loss cannot guarantee delivery. See outcomes, failure boundaries and migration. No release blog or publication is implied.
Task-level MCP mocks - Task YAML accepts mcp_mocks. A task entry replaces the same-named eval-level mock for that task only, a new name adds a mock, and other eval-level mocks still apply. Run, preflight, waza check and result cache keys use each task's effective mocks. Mock fixtures are read only for enabled tasks selected by the run, and an eval-level mock's fixtures only when one of those tasks or configured trigger tests uses it. A task file that already contained an mcp_mocks key, previously ignored with an unknown-field warning, now applies it; an empty task list is rejected. before_run hooks now run before tasks are loaded for MCP admission and before the engine starts, still once per run (once per pass with --baseline), so they can generate task inputs. Admission retains trigger-only dependencies after initialization and each pass's setup. Ctrl+C during before_run, MCP admission or engine initialization now stops the run promptly, and after_run still runs once (bounded to one minute after an interrupt).
Lab conversation viewer - The opt-in Lab inspector shows recorded user and agent messages in saved order without mixing in tool events or inventing message provenance (#753). It displays the first repetition; missing messages remain unavailable.
Program-grader context - WAZA_GRADING_CONTEXT points program graders to a private temporary JSON file containing recorded run evidence, including tool and mocked-command observations (#799). Stdin and WAZA_WORKSPACE_DIR are unchanged; unavailable evidence remains null, and write or cleanup failures fail grading. See the program grader for privacy and same-user trust boundaries.
Host lifecycle evidence - New results use schema 1.8 and can record model-turn boundaries and host shell observations in runs[].host_lifecycle (#816). Only structured native events are projected; missing observations are not inferred, and historical results remain readable.
1.0.28 and bundled CLI to 1.0.105 (#751, #820). Published standalone v0.39.1 bundled SDK 1.0.21 and CLI 1.0.85.fast-uri and site's postcss-selector-parser transitive dependencies (#780).code grader input - The JavaScript wrapper for code graders now reads its grading context directly from standard input (file descriptor 0) instead of through process.stdin, which made the pipe non-blocking. Large contexts, such as long transcripts, no longer fail intermittently with EAGAIN and an exit status 1 grader error on Linux and macOS when the pipe is momentarily empty. Assertions, results and the Python wrapper are unchanged.eval_id values now fail with an explicit diagnostic naming both files instead of silently hiding one result (#880).waza adversarial --artifacts-dir now accepts bare and ./ relative paths with the same current-directory semantics as absolute paths (#878).inputs.environment is reported without exposing values instead of silently ignored (#842); per-task environment support is not added.waza dev accepts workspace skill names, directories, and existing SKILL.md files consistently in iterative, Copilot-report, and trigger-scaffolding modes, including Windows paths. Bare names retain workspace precedence; explicit paths never fall back to another skill. Batch names and --all are unchanged (#747, fixes #746).Full comparison: v0.39.0...v0.39.1. Contributors: Shayne Boyer (@spboyer), Copilot, and Dependabot.
waza new eval --scenario --template repository|cli|mcp scaffolds workflows without a required skill. Scenarios require exact eval schema 2.0, disable ambient skills unless opted in, and bypass result caching; legacy v1 suites retain their behavior (#670).waza preflight checks local prerequisites and named grader references without starting agents or services. Tasks can describe outcome, boundary, recovery, and quality requirements; metadata does not add assertions or prove satisfaction (#671).pass^k curve are preserved and displayed separately from weighted scores. Retry-enabled trial statistics describe reliability with retries (#688).evidence_context. Fresh separately owned judge sessions receive per-grader selections; persisted bundles support saved-result regrading. Explicit credentials and model calls are required; this is not OS sandboxing (#704).waza serve --lab opens saved-result charts, filtering, history, task evidence, descriptive comparisons, and reports without replacing the classic dashboard. Recorded evidence includes reliability, provenance, requirement observations, and prompt-judging metadata (#652, #723).native-copilot planning/collection supports fixed named models, single turns, native text checks, retained attempts/lifecycle/output/usage, and dashboard exports. Durable creation supports Linux/macOS only; skills, client tools, native assurance, runtime-version evidence, and provider currency are unsupported or unavailable as documented. Collection can incur usage; observed budgets are not atomic spending caps (#724).waza assure checks supported native graders against fixed reference cases; separate waza assure calibrate requires explicit paid-call acknowledgment and reviewed inputs. Additional preserved-native and fault/retry components retain separate, non-default API/profile boundaries (#666, #667, #703).is_significant no longer treats a task's non-negative score interval excluding zero as a significance test; the dashboard badge is removed and older results still load (#688).1.0.15 to 1.0.21, OpenTelemetry, Azure Blob Storage SDK, Astro/Starlight, React Query, dashboard/build tooling, and related lockfiles (#712, #721, plus dependency updates in the full comparison).schemaVersion: "2.0"; older executables reject that major. A present scenario must be a nonblank string: remove invalid/null legacy fields or supply the valid scenario contract. Task/results versions evolve separately; new results use 1.7.Full comparison: v0.38.10...v0.39.0. Contributors: Shayne Boyer (@spboyer), Charley Wu (@akunzai), Copilot, and Dependabot. First-time contributor: Charley Wu (@akunzai).
graders[].ref can resolve Go-module-style GitHub references through waza get, with pinned lockfiles, digest verification, offline cache support, and runtime expansion (#15, #494).waza registry search and waza registry add for discovering shared graders and adding compatible references to eval files (#17, #493).dorny/paths-filter to v4 for current runner support and merge-queue handling (#519).tool_calls graders now use canonical tool_events[].args data, preserving MCP and custom tool arguments across results.json round trips (#474, #477).go-runewidth (#495, #497, #498, #499, #500, #502, #503).gh release create.waza update now passes the PowerShell installer URL reliably, preventing Invoke-RestMethod from failing with a null or empty Uri (#448).json_schema graders now accept the documented extract_json option in eval and task YAML, extracting exactly one JSON object or array candidate before schema validation (#457).config.mcp_servers entries now default omitted tools to ["*"], matching MCP mock behavior so configured stdio and HTTP servers expose their tools to Copilot SDK evaluations (#449).mcp_mocks now explicitly exposes declared tools to the bundled Copilot CLI, preventing valid mock servers from being skipped during Copilot SDK evaluations (#440).extract_json now rejects output containing a JSON code block plus any additional JSON document.json_schema grader can now extract exactly one JSON document from prose or a Markdown JSON code block with extract_json: true, while retaining strict pure-JSON validation by default.waza suggest now supports targeted generation with --count, --focus, --dry-run, --apply, and --force (#357, #380)checkpoints[] and on_failure policies (#358, #386)waza rubric subcommand ships in this release (#360, #381)waza spec verify to report eval coverage against SKILL.md requirements (#361, #385)waza run --otel-exporter, --otel-endpoint, --otel-headers, --otel-file, and --otel-include-payloads (#362, #383)mcp_mocks: for hermetic Copilot SDK tool-call evals (#363, #387)waza gate with stable exit codes for pass, regression, golden failure, and config errors (#364, #384)waza adversarial and eval-level adversarial: pack configuration for prompt-injection and scope-bypass checks (#365, #392)tool_events[]; tool graders can assert structured argument matchers through expect_tools[].args and tool_calls.expect[].args (#366, #388)waza run --snapshot and waza replay with a self-contained snapshot artifact format (#367, #391)schemaVersion compatibility for public artifacts (#368, #382)Last-Event-ID / lastEventId resume support for dashboard event streams, including legacy /api/events (#178, #397)waza suggest engine failures — Engine failures are now surfaced by waza suggest instead of being hidden behind success-shaped output (#330)WAZA_PROMPT_GRADER_TIMEOUT (#319)github.com/github/copilot-sdk/go to v1.0.0 and surfaced premium-request credits on the dashboard (#311)--model startup arg — The Copilot CLI validates the startup --model flag against the Copilot catalog before BYOK provider config is applied, so provider-only model IDs would fail. The --model startup arg is now skipped when a BYOK provider is configured (#305, #306)--model is now passed via CLIArgs so it correctly overrides user settings and experiment flights (#263)PATH when the bundled binary is unavailable (#300)AgentEngine cancellation handling around caller contexts to make shutdown semantics more predictable (#290)waza update command — Added an update command for upgrading local Waza installations (#288)skill_invocation graders can now assert that specific skills must not be invoked (#286, #291)waza run now respects cancellation signals more reliably (#279)waza run now reuses a shared Copilot client and auto-sizes parallel workers when --workers is unset (Note truncated.
The standalone release and separately versioned azd extension are published. See the release article and download reference.
waza coverage counts selected task files or CSV
rows through the execution task loader; equivalent glob and explicit selectors
agree, and an empty selection reports zero tasks with Partial coverage
(#897).waza tokens check --strict returns exit 1 for
exceeded budgets while advisory exceedance remains exit 0 and invalid input
remains exit 2 (#896).config.group_by: tags resolves each task's complete
ordered tag set into the existing group field; tagged runs produce CLI and
JSON group statistics and untagged tasks remain ungrouped
(#891).inputs,
reject rendered-empty commands and invalidate legacy task caches when hook
configuration or inputs change
(#890)../ relative and
absolute --artifacts-dir paths execute the same selected pack
(#887).code grader input - The JavaScript wrapper reads grading
context directly from file descriptor 0, avoiding intermittent EAGAIN
failures for large contexts on Linux and macOS. Assertions, results and the
Python wrapper are unchanged
(#882).Full comparison: v0.40.0...v0.40.1.
Run an exact work unit: Negotiate runner capabilities, validate a captured shard/task/trial/attempt and receive an input-derived identity and a struct
contains assertion, not an entire task or
suite. Unsupported selected semantics fail rather than being approximated.The public image reference is:
ghcr.io/microsoft/waza-runner-reference@sha256:ded068a636dc721b8e38dc2a68bcbcbdb0d4b33ce5da8d182d562d189a712a39
Its source is commit 93e80d7;
see the publisher
and anonymous-digest verifier.
Only linux/amd64 is published and verified for this image.
mcp_mocks key that older versions ignored now takes effect.
before_run also runs before task loading and MCP admission, so review
existing task files and hooks when upgrading.0.40.0. The immutable tag's version.txt and
extension.yaml still say 0.39.1 because the established release workflow
synchronizes those files afterward on main. For a versioned source build use
make build VERSION=0.40.0; later synchronization does not alter this tag.dev; identify this image by its immutable digest and source
provenance rather than its binary version string.1.0.28 and bundled CLI is 1.0.105.Full Changelog: https://github.com/microsoft/waza/compare/v0.39.1...v0.40.0
contains assertion, not an entire task orThe public image reference is:
ghcr.io/microsoft/waza-runner-reference@sha256:ded068a636dc721b8e38dc2a68bcbcbdb0d4b33ce5da8d182d562d189a712a39
Its source is commit 93e80d7;
see the publisher
and anonymous-digest verifier.
Only linux/amd64 is published and verified for this image.
mcp_mocks key that older versions ignored now takes effect.before_run also runs before task loading and MCP admission, so review0.40.0. The immutable tag's version.txt andextension.yaml still say 0.39.1 because the established release workflowmake build VERSION=0.40.0; later synchronization does not alter this tag.dev; identify this image by its immutable digest and source1.0.28 and bundled CLI is 1.0.105.Full Changelog: v0.39.1...v0.40.0
The Waza azd extension v0.40.0 is published with six platform-specific
archives. Its source is commit
93e80d7.
The generated extension registry entry is synchronized to main by
#883. All six archive URLs,
entrypoints and SHA-256 values were checked against the published assets;
strict registry validation passed with the existing missing-schemaVersion
warning. The macOS/arm64 archive was downloaded, checksum-verified and executed
and reports waza version 0.40.0. Registry-based installation through azd has
not been exercised; the other platform archives were not executed locally.
The immutable release tag retains the earlier source version metadata, while
these published extension archives report 0.40.0. Later main-branch timeout and
cancellation fixes are Unreleased and are not included in these archives.
See the Waza v0.40.0 release notes
for features, compatibility notes, known limitations, the verified runner image,
and contributor credits.
The standalone release is published. Extension artifacts and registry synchronization are independent; see release availability. The exact-unit runner controls do not fix the broader legacy multi-trial timeout issue #865. The later timeout/persistence fix #870 is post-v0.40.0 source behavior, not included in the published v0.40.0 artifacts.
Opt-in exact Copilot/text runner - Add negotiated inline.copilot-text execution through the owned production SDK engine for one declared shard/task/trial/attempt (#758). Requires explicit model and paid consent; tools and skills are disabled and general native file/tool/skill workflows remain unsupported. Paid HTTP execution additionally requires a loopback peer, localhost/loopback Host and same-origin browser requests; remote mock and offline routes retain their previous behavior. Admitted eval metrics are identity-bound, uncomputed metadata, not aggregate values or threshold decisions. Cancellation and effective execution/shutdown deadlines interrupt the private transport; lifecycle and incomplete-usage diagnostics cannot become success, and isolated sessions never resume with unrestricted defaults solely to retrieve usage. CLI/API, terminal schema and Lab distinguish copilot from mock. Existing mock requests, identity algorithm, local adapters and result formats are unchanged; older servers reject the new capability before execution. SDK-fake conformance is not live-provider or native-platform evidence. See invocation and migration.
Selected AgentV assertion interoperability - Add offline, version-pinned waza interop import/export for one explicitly selected case-sensitive contains assertion and its single-check native grader mapping (#789). Retained upstream fixture/oracle controls verify binary decisions and scores; unsupported selected semantics fail explicitly and excluded task/suite semantics are reported. This is not whole-task/suite support: AgentEvals-spec, Harbor, ASSERT and full evidence/provenance round trips remain unsupported, and #789 remains open. See the assertion guide.
Runner container references - Keep the default scratch reference target mock/text-only and add an explicit digest-pinned Debian/glibc native-reference target for the exact-unit native Copilot text profile (#762). The real bundled CLI is exercised against an in-container synthetic provider with runtime networking disabled, alongside positive discovery/hook controls, restriction checks, exact selection, timeout, cancellation and process cleanup. Docker CI requires native Linux AMD64; local ARM64 or emulated runs are not equivalent. Local image IDs are not registry pins. The published image record identifies the immutable image digest, source, platform and successful publisher/verifier: anonymous pull, signed provenance/finalization and mock plus bundled-native synthetic-provider production controls passed (1/1 test, 0 skipped). Infrastructure-owner review was waived for this one publication, not performed. This is not live-provider reliability or model-quality evidence. No scheduler or Lab change is introduced. See the quickstart and maintainer update policy.
Bounded local compatibility - Add offline runner adapt and opt-in run --runner-compat for one literal mock/text task, trial and attempt (#761). Both use the byte-preserving adapter; local execution uses the shared runner service, current EvaluationOutcome writer schema 1.8 and local 0/1/2 exits. Ordinary run, historical schemas and artifact commands remain available; none are deprecated. Unsupported mappings, flags, ambient configuration, fixtures and adjacent trigger work fail explicitly. Remove --runner-compat to retain native/local behavior, or explicitly choose terminal JSON with runner run. No Lab change is required because no new result evidence/status is introduced. See the compatibility and migration table.
Runner negotiation 1.0 - Add offline JSON stdin/stdout runner capabilities and runner negotiate, matching HTTP APIs, strict request/response schemas, conformance fixtures and live Lab incompatibility diagnostics (#757). Compatibility alone never starts work.
Exact captured mock/text work units - Add runner validate / runner run and HTTP equivalents with fail-closed shard/task/trial/attempt admission, one fresh engine call, no task scanning, cache or automatic retries, and live Lab selection validation (#758). Each explicit repeat executes again, not deduplication. Opt-in Copilot/text is described above; general native file/tool/skill workflows remain unavailable. Existing local commands and artifacts are unchanged; no forms are deprecated. See the contract and migration guide.
Input-derived runner identities - Admitted validation/execution receipts now carry identity with version sha256-waza-c14n-v1, canonical input digest and run/shard/task/trial/attempt lineage through CLI JSON, HTTP, Lab and validation export (#759). Independent frozen source and producer vectors define the stability boundary. Invalid/unadmitted work has no identity; later execution failures retain it without implying execution success. Historical local results and receipts without identities are unchanged and are not backfilled. The unshipped sha256-json-v1 draft is not compatible; compare version and ID together. See normalization, conformance and migration.
Runner terminal envelope - runner run and HTTP /runner/run now emit one schema-validated waza.runner-terminal envelope, with success/evaluation/infrastructure/cancellation/timeout/unsupported/invalid exits 0–6, actual admitted identity and bounded observed verdicts (#760). This replaces only the earlier unshipped run receipt, not offline negotiation/validation or local result artifacts. Lab's explicit mock-run action shows and exports the real response; edits and errors clear stale success. Short/failed output never retries, and SIGKILL/host loss cannot guarantee delivery. See outcomes, failure boundaries and migration.
Task-level MCP mocks - Task YAML accepts mcp_mocks. A task entry replaces the same-named eval-level mock for that task only, a new name adds a mock, and other eval-level mocks still apply. Run, preflight, waza check and result cache keys use each task's effective mocks. Mock fixtures are read only for enabled tasks selected by the run, and an eval-level mock's fixtures only when one of those tasks or configured trigger tests uses it. A task file that already contained an mcp_mocks key, previously ignored with an unknown-field warning, now applies it; an empty task list is rejected. before_run hooks now run before tasks are loaded for MCP admission and before the engine starts, still once per run (once per pass with --baseline), so they can generate task inputs. Admission retains trigger-only dependencies after initialization and each pass's setup. Ctrl+C during before_run, MCP admission or engine initialization now stops the run promptly, and after_run still runs once (bounded to one minute after an interrupt).
Lab conversation viewer - The opt-in Lab inspector shows recorded user and agent messages in saved order without mixing in tool events or inventing message provenance (#753). It displays the first repetition; missing messages remain unavailable.
Program-grader context - WAZA_GRADING_CONTEXT points program graders to a private temporary JSON file containing recorded run evidence, including tool and mocked-command observations (#799). Stdin and WAZA_WORKSPACE_DIR are unchanged; unavailable evidence remains null, and write or cleanup failures fail grading. See the program grader for privacy and same-user trust boundaries.
Host lifecycle evidence - New results use schema 1.8 and can record model-turn boundaries and host shell observations in runs[].host_lifecycle (#816). Only structured native events are projected; missing observations are not inferred, and historical results remain readable.
1.0.28 and bundled CLI to 1.0.105 (#751, #820). Published standalone v0.39.1 bundled SDK 1.0.21 and CLI 1.0.85.fast-uri and site's postcss-selector-parser transitive dependencies (#780).inputs.environment is reported without exposing values instead of silently ignored (#842); per-task environment support is not added.Full comparison: v0.39.1...v0.40.0.
A failed evaluation should leave you with a useful next step: identify the work that ran, understand whether a check failed or the runtime failed, and inspect the evidence. That gets harder when a caller silently discovers more tasks, retries work, or treats a successful HTTP response as a passing evaluation.
Waza v0.40.0 makes that boundary explicit. Its new runner accepts one declared text attempt, gives admitted input a deterministic identity, and returns a structured terminal outcome. Alongside it, task-level MCP mocks and richer grading and conversation evidence help you design and investigate ordinary evaluations.
Released November 4, 2026 , this version builds on v0.39.0’s scenario-first workflow and the v0.39.1 skill-target fix. Existing local commands and historical results remain supported; the runner is an additional, deliberately bounded interface, not a requirement to migrate.
Section titled “Execute one attempt, not an implicit suite”
The runner’s unit of work is an explicit shard, task, trial and attempt . It receives captured eval and task text rather than discovering a suite from the caller’s filesystem. Capability negotiation lets a caller find out whether its requested profile is supported before starting a runtime.
From a checkout of the v0.40.0 tag, using the matching installed binary or a tagged source build , try the credential-free example:
Terminal window
waza runner capabilities
waza runner validate < examples/runner/unit.json
waza runner run < examples/runner/unit.json
capabilities advertises the implemented profiles. validate checks admission without constructing an engine. Only run starts the selected runtime.
The checked-in request selects {"shard":"shard-a","task":"selected","trial":2,"attempt":2} . Its eval allows three trials and three attempts, but 2/2 are selectors, not loop counts . Waza executes just that attempt with one fresh engine call: no neighboring tasks, glob expansion, cached replay or automatic retries. This mock example returns a passed text check; it proves harness wiring, not agent quality. Explicitly dispatching it again executes again, with no cross-worker deduplication or exactly-once side-effect guarantee.
The shipped profiles are mock/text and explicitly opted-in Copilot/text . Copilot requires a model and paid-execution consent, with tools and skills disabled. General native file/tool/skill workflows remain unsupported. The runner contract explains admission, deadlines, CLI/HTTP APIs and the additional local-only native HTTP policy. Keep ordinary waza run for broader workflows.
Section titled “Know which input a result belongs to”
An admitted unit carries a canonical digest and logical run / shard / task / trial / attempt identities, using the versioned sha256-waza-c14n-v1 algorithm. Validation and execution derive them from the same original input, so a caller can connect a validation receipt to its execution result without depending on timestamps, machine paths or model output.
Presentation-only changes such as mapping key order or comments normalize. Meaningful distinctions such as absent, null, empty, zero and false remain distinct. Changing the attempt changes its attempt identity; changing the eval changes the whole hierarchy. The identity guide and frozen vectors define the precise stability rules.
These are input-derived identities, not execution attestations . Offline validation has zero engine calls even when it returns an identity. A later runtime failure retains that identity; repeated execution of the same input uses the same IDs. Neither a hash nor an ID certifies success, trusted provenance, isolation or delivery.
Section titled “Handle failed checks differently from failed infrastructure”
runner run returns one waza.runner-terminal envelope with a stable outcome and exit category. A caller can distinguish a failed assertion from a request it should correct, or a runtime failure it should investigate.
success (exit 0) : All admitted checks passed in a complete observed attempt.
evaluation_failure (exit 1) : At least one observed check failed.
infrastructure_failure (exit 2) : Startup, execution, cleanup or output failed.
cancelled (exit 3) : Cancellation was observed.
timeout (exit 4) : An applicable deadline expired.
unsupported_capability (exit 5) : Required runner version or capability is unavailable.
invalid_input (exit 6) : The request or selected input is invalid.
Inspect outcome , exit_code , diagnostics and the recorded failure boundary, not just HTTP status: a complete HTTP runner request can return 200 even when execution fails . The bounded observation summarizes the selected attempt’s actual checks; it is not a full transcript, aggregate evaluation score or billing report. Missing usage is unavailable, not zero.
The terminal guide covers response fields and transport failures. One emission attempt cannot guarantee receipt after a broken pipe, process crash or uncatchable termination.
Section titled “Use the same boundary locally or in a container”
For a narrow local mock/text suite, runner adapt captures explicitly named sources offline. Opt-in run --runner-compat uses the shared service while retaining local result JSON and its existing 0/1/2 exit policy, rather than replacing it with terminal JSON. The adapter supports one literal task, one trial and one attempt; unsupported configuration is an error, not a silent fallback. Broader adaptation remains deferred under #761 . The compatibility table helps choose the interface without deleting meaningful configuration to fit it.
The container reference exercises the same exact-unit contract. Its default scratch target is mock-only; the explicit Debian/glibc native reference adds the bundled Copilot runtime for the restricted text profile. The published native image passed anonymous, digest-pinned acceptance on Linux AMD64 , with mock and bundled-native synthetic-provider controls. That is not live-provider reliability, model-quality evidence or support for ARM64 and Windows. The image reports dev , not the standalone binary’s 0.40.0 ; identify it by its immutable digest and source.
Use the container guide’s published-image record for the exact digest, provenance, acceptance results and one-publication infrastructure-review waiver. Publication, automated acceptance and independent review are separate facts.
Section titled “Design more focused tests and inspect better evidence”
These additions also improve ordinary evaluations, independently of adopting the runner:
Change one task’s MCP response. Task-level MCP mocks replace same-named eval mocks for that task only; other eval mocks still apply. One case can approve a question while another declines it. They require the Copilot SDK executor, not the mock executor. Finite MCP sequences now continue across follow-up turns within an attempt, while each task, trial and retry starts fresh.
Give an external grader recorded context. Program graders receive a private JSON file through WAZA_GRADING_CONTEXT , including recorded tool and mocked-command observations. Existing stdin output and WAZA_WORKSPACE_DIR stay unchanged. Write and cleanup failures fail grading; the program and same-user processes remain trusted.
Check what the host actually recorded. Host lifecycle evidence adds structured model-turn and shell observations. A missing shell exit code is unknown, not zero; command output text is not a substitute for a host event. New results use schema 1.8 , independently of runner request/terminal 1.0 .
Read a saved conversation without tool-event noise. Lab’s Conversation tab shows recorded user and agent text from the first repetition in saved order. Missing messages are not reconstructed; Trajectory retains the underlying events. The Dashboard Lab article introduces the workspace.
There is also a small, useful interoperability step: waza interop import/export converts one explicitly selected, pinned AgentV case-sensitive contains assertion to or from a single native text check, offline. The assertion adapter guide explains the excluded semantics. This is not a task or suite import: whole-task/suite interoperability and full evidence/provenance round trips remain unsupported under #789 .
Section titled “Upgrade with the published release’s limits in view”
Install the standalone v0.40.0 release or run waza update ; no source build is required. The download and extension reference lists platform assets and the separately published azd extension. Source builders should use make build VERSION=0.40.0 : the immutable tag retains older metadata because version-file synchronization happens after tagging. The release includes Copilot SDK 1.0.28 and bundled CLI 1.0.105 .
Known limitations in the published archives: exact-unit runner controls do not fix the separate legacy multi-trial timeout issue #865 , which can prevent ordinary multi-trial runs from retaining completed results. The later timeout/persistence fix #870 is post-v0.40.0 source behavior, not included in the published v0.40.0 artifacts. The JavaScript code grader can also fail intermittently with EAGAIN on large stdin contexts on Linux/macOS; its fix #882 is main-only and unreleased , as is the subsequent cancellation-cleanup follow-up recorded in the Unreleased changelog . Those main changes have not been republished as v0.40.0.
See the full release comparison for the technical breakdown. Start with offline validation and the mock example; then opt into native text execution only when its scope, credentials and possible cost match the experiment you intend to run.
Nothing published for this version
This patch fixes single-skill targeting in waza dev and includes the documentation-site updates merged since v0.39.0.
This patch fixes single-skill targeting in waza dev and includes the documentation-site updates merged since v0.39.0.
waza dev SKILL.md treated the file as a directory. You can now pass a skill name, directory, or SKILL.md file to iterative, Copilot-report, and trigger-scaffolding modes, including relative and absolute paths on Windows../directory or .\directory to select a path explicitly; batch names and --all retain their behavior.waza dev .\skills\my-skill\SKILL.mdThe directory forms waza dev . and waza dev .\skills\my-skill continue to work.
dev inputs before all three single-skill modes, reject unrelated files with contextual errors, and add native-platform regression coverage, including Windows malformed-path handling (#747, fixes #746).Full comparison: v0.39.0...v0.39.1.
Contributors: Shayne Boyer (@spboyer), Copilot, and Dependabot.
This extension packages Waza v0.39.1, including the single-skill file-target fix.
azd waza dev accepts a workspace skill name, skill directory, or existing SKILL.md file consistently across iterative, Copilot-report, and trigger-scaffolding modes, including Windows paths (#747, fixes #746).--all keep their behavior.azd waza dev .\skills\my-skill\SKILL.mdThe documentation assistant's availability depends on the deployed site configuration.
The six platform archives are published with this release. The custom registry on main receives version 0.39.1 when the release version-sync PR merges; publishing these archives alone does not update that registry.
Full comparison: v0.39.0...v0.39.1.
Contributors: Shayne Boyer (@spboyer), Copilot, and Dependabot.
waza dev accepts workspace skill names, directories, and existing SKILL.md files consistently in iterative, Copilot-report, and trigger-scaffolding modes, including Windows paths. Bare names retain workspace precedence; explicit paths never fall back to another skill. Batch names and --all are unchanged (#747, fixes #746).Full comparison: v0.39.0...v0.39.1. Contributors: @spboyer, Copilot, and Dependabot.
Nothing published for this version
Nothing published for this version
Scenario-first evaluation, agent-free preflight, deterministic recovery fixtures, recorded evidence, independent judging, and opt-in release compariso
Scenario-first evaluation, agent-free preflight, deterministic recovery fixtures, recorded evidence, independent judging, and opt-in release comparisons.
waza new eval --scenario --template repository|cli|mcp scaffolds workflows without a required skill. Scenarios require exact eval schema 2.0, disable ambient skills unless opted in, and bypass result caching; legacy v1 suites retain their behavior (#670).waza preflight checks local prerequisites and named grader references without starting agents or services. Tasks can describe outcome, boundary, recovery, and quality requirements; metadata does not add assertions or prove satisfaction (#671).pass^k curve are preserved and displayed separately from weighted scores. Retry-enabled trial statistics describe reliability with retries (#688).evidence_context. Fresh separately owned judge sessions receive per-grader selections; persisted bundles support saved-result regrading. Explicit credentials and model calls are required; this is not OS sandboxing (#704).waza serve --lab opens saved-result charts, filtering, history, task evidence, descriptive comparisons, and reports without replacing the classic dashboard. Recorded evidence includes reliability, provenance, requirement observations, and prompt-judging metadata (#652, #723).native-copilot planning/collection supports fixed named models, single turns, native text checks, retained attempts/lifecycle/output/usage, and dashboard exports. Durable creation supports Linux/macOS only; skills, client tools, native assurance, runtime-version evidence, and provider currency are unsupported or unavailable as documented. Collection can incur usage; observed budgets are not atomic spending caps (#724).waza assure checks supported native graders against fixed reference cases; separate waza assure calibrate requires explicit paid-call acknowledgment and reviewed inputs. Additional preserved-native and fault/retry components retain separate, non-default API/profile boundaries (#666, #667, #703).is_significant no longer treats a task's non-negative score interval excluding zero as a significance test; the dashboard badge is removed and older results still load (#688).1.0.15 to 1.0.21, OpenTelemetry, Azure Blob Storage SDK, Astro/Starlight, React Query, dashboard/build tooling, and related lockfiles (#712, #721, plus dependency updates in the full comparison).schemaVersion: "2.0"; older executables reject that major. A present scenario must be a nonblank string: remove invalid/null legacy fields or supply the valid scenario contract. Task/results versions evolve separately; new results use 1.7.Full comparison: v0.38.10...v0.39.0. Contributors: Shayne Boyer (@spboyer), Charley Wu (@akunzai), Copilot, and Dependabot. First-time contributor: Charley Wu (@akunzai).
Nothing published for this version
Nothing published for this version
Go implementation of waza CLI
Go implementation of waza CLI
All notable changes to waza will be documented in this file.
The format is based on Keep a Changelog,
and this project adheres to Semantic Versioning.
graders[].ref can resolve Go-module-style GitHub references through waza get, with pinned lockfiles, digest verification, offline cache support, and runtime expansion (#15, #494).waza registry search and waza registry add for discovering shared graders and adding compatible references to eval files (#17, #493).dorny/paths-filter to v4 for current runner support and merge-queue handling (#519).tool_calls graders now use canonical tool_events[].args data, preserving MCP and custom tool arguments across results.json round trips (#474, #477).go-runewidth (#495, #497, #498, #499, #500, #502, #503).gh release create.waza update now passes the PowerShell installer URL reliably, preventing Invoke-RestMethod from failing with a null or empty Uri (#448).json_schema graders now accept the documented extract_json option in eval and task YAML, extracting exactly one JSON object or array candidate before schema validation (#457).config.mcp_servers entries now default omitted tools to ["*"], matching MCP mock behavior so configured stdio and HTTP servers expose their tools to Copilot SDK evaluations (#449).mcp_mocks now explicitly exposes declared tools to the bundled Copilot CLI, preventing valid mock servers from being skipped during Copilot SDK evaluations (#440).extract_json now rejects output containing a JSON code block plus any additional JSON document.json_schema grader can now extract exactly one JSON document from prose or a Markdown JSON code block with extract_json: true, while retaining strict pure-JSON validation by default.waza suggest now supports targeted generation with --count, --focus, --dry-run, --apply, and --force (#357, #380)checkpoints[] and on_failure policies (#358, #386)waza rubric subcommand ships in this release (#360, #381)waza spec verify to report eval coverage against SKILL.md requirements (#361, #385)waza run --otel-exporter, --otel-endpoint, --otel-headers, --otel-file, and --otel-include-payloads (#362, #383)mcp_mocks: for hermetic Copilot SDK tool-call evals (#363, #387)waza gate with stable exit codes for pass, regression, golden failure, and config errors (#364, #384)waza adversarial and eval-level adversarial: pack configuration for prompt-injection and scope-bypass checks (#365, #392)tool_events[]; tool graders can assert structured argument matchers through expect_tools[].args and tool_calls.expect[].args (#366, #388)waza run --snapshot and waza replay with a self-contained snapshot artifact format (#367, #391)schemaVersion compatibility for public artifacts (#368, #382)Last-Event-ID / lastEventId resume support for dashboard event streams, including legacy /api/events (#178, #397)waza suggest engine failures — Engine failures are now surfaced by waza suggest instead of being hidden behind success-shaped output (#330)WAZA_PROMPT_GRADER_TIMEOUT (#319)github.com/github/copilot-sdk/go to v1.0.0 and surfaced premium-request credits on the dashboard (#311)--model startup arg — The Copilot CLI validates the startup --model flag against the Copilot catalog before BYOK provider config is applied, so provider-only model IDs would fail. The --model startup arg is now skipped when a BYOK provider is configured (#305, #306)--model is now passed via CLIArgs so it correctly overrides user settings and experiment flights (#263)PATH when the bundled binary is unavailable (#300)AgentEngine cancellation handling around caller contexts to make shutdown semantics more predictable (#290)waza update command — Added an update command for upgrading local Waza installations (#288)skill_invocation graders can now assert that specific skills must not be invoked (#286, #291)waza run now respects cancellation signals more reliably (#279)waza run now reuses a shared Copilot client and auto-sizes parallel workers when --workers is unset (#135, #221)Note: This release includes the changes previously prepared under 0.32.0, which was not published.
.waza.yaml can now configure files.evalFile, files.taskGlob, and files.taskFileSuffix, with the new naming carried through scaffolding, workspace discovery, discovery mode, schemas, and docs while preserving the existing eval.yaml and tasks/*.yaml defaults (#254, closes #232)config.instruction_files and task-level instruction_files now copy files from the active context into task workspaces and append path-labeled contents to the Copilot system message (#248, closes #239)CopilotEngine instead of constructing a Copilot client directly, keeping grader execution aligned with engine configuration and preserving follow-up recovery behavior (#258, closes #54)copilot-cli bundles are updated from 1.0.2 to 1.0.49 across supported platforms, with reproducible pinned bundle generation via COPILOT_CLI_VERSION (#260, closes #244)waza new skill no longer asks for a nonstandard skill type or emits type: frontmatter, and the wizard now rejects early exits that omit required name or description fields (#261, closes #243)waza check eval discovery — Nested skills and separated evals are discovered consistently in multi-skill workspaces (#247, closes #238)SKILL.md body sections as well as frontmatter descriptions (#236, closes #223)github.com/github/copilot-sdk/go to v0.3.0, migrated session event handling to typed payloads, and refreshed transcript, logging, web API, usage collection, suggestion trace, and test coverage for the new API (#255, closes #253)go install guidance and clarified Windows/WSL install behavior (#246, closes #242; #245, closes #241).agent.md) eval support — Discover .agent.md files alongside SKILL.md, parse agent-specific frontmatter (tools, model, handoffs, mcp-servers, agents), auto-inject tool_constraint grader from agent tools: field, complete worked example under examples/custom-agent/, and new "Evaluating Custom Agents" docs guide (#226, closes #225)_output_contains expectations against file contents now work in CI without a real model. Mock response includes task metadata, file paths, and a 1KB content preview per resource (#228, closes #227)waza serve no longer crashes when stdin isn't a terminal — MCP stdio server only starts when term.IsTerminal() is true; piped input or background mode no longer kills the HTTP dashboard (#224)BenchmarkSpec → EvalSpec, TestRunner → EvalRunner. Not a breaking change for external consumers (types live in internal/) (#222).agent.md coverage to quickstart, getting-started, GUIDE, TUTORIAL, examples README; updated mock engine descriptions in INTEGRATION-TESTING and eval-yaml guide (#230)waza quality command — LLM-as-Judge skill quality scoring that evaluates skill output quality using a configurable judge model (#218)waza check now includes an advisory that flags skills with overly broad scope, helping authors tighten skill definitions (#219)--keep-workspace flag — Preserve the temporary workspace after task execution for debugging agent output (#123, #217)--no-skills flag and disabled_skills config — Disable specific skills during evaluation to isolate behavior (#126, #216)skill_directories — Specify different skill directories for individual tasks in eval YAML (#156, #215)waza models command — List all available models supported by the configured engine (#208)TestCase definitions are now properly rejected (#132, #206)output_contains_any expectation — New expectation field that passes when the agent response contains any one of the specified strings (#203)max_response_time_ms behavior rule — Enforce maximum response time constraints on agent execution (#201)prompt field can now reference an external file path instead of inline text (#157, #200)tool_calls grader — New grader type that validates the specific tool calls an agent makes during execution (#187, #202)run --output-dir now groups result files by timestamp for cleaner organization (#153)--discover finds eval.yaml in nested layout — Skill discovery now correctly locates eval.yaml files in evals/{name}/ directories at the project root (#44)Note truncated.
Nothing published for this version
chore: Release v0.38.8 — registry and version sync by @spboyer in https://github.com/microsoft/waza/pull/629
Full Changelog: https://github.com/microsoft/waza/compare/v0.38.8...v0.38.9
Full Changelog: v0.38.8...v0.38.9
All notable changes to waza will be documented in this file.
The format is based on Keep a Changelog,
and this project adheres to Semantic Versioning.
graders[].ref can resolve Go-module-style GitHub references through waza get, with pinned lockfiles, digest verification, offline cache support, and runtime expansion (#15, #494).waza registry search and waza registry add for discovering shared graders and adding compatible references to eval files (#17, #493).dorny/paths-filter to v4 for current runner support and merge-queue handling (#519).tool_calls graders now use canonical tool_events[].args data, preserving MCP and custom tool arguments across results.json round trips (#474, #477).go-runewidth (#495, #497, #498, #499, #500, #502, #503).gh release create.waza update now passes the PowerShell installer URL reliably, preventing Invoke-RestMethod from failing with a null or empty Uri (#448).json_schema graders now accept the documented extract_json option in eval and task YAML, extracting exactly one JSON object or array candidate before schema validation (#457).config.mcp_servers entries now default omitted tools to ["*"], matching MCP mock behavior so configured stdio and HTTP servers expose their tools to Copilot SDK evaluations (#449).mcp_mocks now explicitly exposes declared tools to the bundled Copilot CLI, preventing valid mock servers from being skipped during Copilot SDK evaluations (#440).extract_json now rejects output containing a JSON code block plus any additional JSON document.json_schema grader can now extract exactly one JSON document from prose or a Markdown JSON code block with extract_json: true, while retaining strict pure-JSON validation by default.waza suggest now supports targeted generation with --count, --focus, --dry-run, --apply, and --force (#357, #380)checkpoints[] and on_failure policies (#358, #386)waza rubric subcommand ships in this release (#360, #381)waza spec verify to report eval coverage against SKILL.md requirements (#361, #385)waza run --otel-exporter, --otel-endpoint, --otel-headers, --otel-file, and --otel-include-payloads (#362, #383)mcp_mocks: for hermetic Copilot SDK tool-call evals (#363, #387)waza gate with stable exit codes for pass, regression, golden failure, and config errors (#364, #384)waza adversarial and eval-level adversarial: pack configuration for prompt-injection and scope-bypass checks (#365, #392)tool_events[]; tool graders can assert structured argument matchers through expect_tools[].args and tool_calls.expect[].args (#366, #388)waza run --snapshot and waza replay with a self-contained snapshot artifact format (#367, #391)schemaVersion compatibility for public artifacts (#368, #382)Last-Event-ID / lastEventId resume support for dashboard event streams, including legacy /api/events (#178, #397)waza suggest engine failures — Engine failures are now surfaced by waza suggest instead of being hidden behind success-shaped output (#330)WAZA_PROMPT_GRADER_TIMEOUT (#319)github.com/github/copilot-sdk/go to v1.0.0 and surfaced premium-request credits on the dashboard (#311)--model startup arg — The Copilot CLI validates the startup --model flag against the Copilot catalog before BYOK provider config is applied, so provider-only model IDs would fail. The --model startup arg is now skipped when a BYOK provider is configured (#305, #306)--model is now passed via CLIArgs so it correctly overrides user settings and experiment flights (#263)PATH when the bundled binary is unavailable (#300)AgentEngine cancellation handling around caller contexts to make shutdown semantics more predictable (#290)waza update command — Added an update command for upgrading local Waza installations (#288)skill_invocation graders can now assert that specific skills must not be invoked (#286, #291)waza run now respects cancellation signals more reliably (#279)waza run now reuses a shared Copilot client and auto-sizes parallel workers when --workers is unset (#135, #221)Note: This release includes the changes previously prepared under 0.32.0, which was not published.
.waza.yaml can now configure files.evalFile, files.taskGlob, and files.taskFileSuffix, with the new naming carried through scaffolding, workspace discovery, discovery mode, schemas, and docs while preserving the existing eval.yaml and tasks/*.yaml defaults (#254, closes #232)config.instruction_files and task-level instruction_files now copy files from the active context into task workspaces and append path-labeled contents to the Copilot system message (#248, closes #239)CopilotEngine instead of constructing a Copilot client directly, keeping grader execution aligned with engine configuration and preserving follow-up recovery behavior (#258, closes #54)copilot-cli bundles are updated from 1.0.2 to 1.0.49 across supported platforms, with reproducible pinned bundle generation via COPILOT_CLI_VERSION (#260, closes #244)waza new skill no longer asks for a nonstandard skill type or emits type: frontmatter, and the wizard now rejects early exits that omit required name or description fields (#261, closes #243)waza check eval discovery — Nested skills and separated evals are discovered consistently in multi-skill workspaces (#247, closes #238)SKILL.md body sections as well as frontmatter descriptions (#236, closes #223)github.com/github/copilot-sdk/go to v0.3.0, migrated session event handling to typed payloads, and refreshed transcript, logging, web API, usage collection, suggestion trace, and test coverage for the new API (#255, closes #253)go install guidance and clarified Windows/WSL install behavior (#246, closes #242; #245, closes #241).agent.md) eval support — Discover .agent.md files alongside SKILL.md, parse agent-specific frontmatter (tools, model, handoffs, mcp-servers, agents), auto-inject tool_constraint grader from agent tools: field, complete worked example under examples/custom-agent/, and new "Evaluating Custom Agents" docs guide (#226, closes #225)_output_contains expectations against file contents now work in CI without a real model. Mock response includes task metadata, file paths, and a 1KB content preview per resource (#228, closes #227)waza serve no longer crashes when stdin isn't a terminal — MCP stdio server only starts when term.IsTerminal() is true; piped input or background mode no longer kills the HTTP dashboard (#224)BenchmarkSpec → EvalSpec, TestRunner → EvalRunner. Not a breaking change for external consumers (types live in internal/) (#222).agent.md coverage to quickstart, getting-started, GUIDE, TUTORIAL, examples README; updated mock engine descriptions in INTEGRATION-TESTING and eval-yaml guide (#230)waza quality command — LLM-as-Judge skill quality scoring that evaluates skill output quality using a configurable judge model (#218)waza check now includes an advisory that flags skills with overly broad scope, helping authors tighten skill definitions (#219)--keep-workspace flag — Preserve the temporary workspace after task execution for debugging agent output (#123, #217)--no-skills flag and disabled_skills config — Disable specific skills during evaluation to isolate behavior (#126, #216)skill_directories — Specify different skill directories for individual tasks in eval YAML (#156, #215)waza models command — List all available models supported by the configured engine (#208)TestCase definitions are now properly rejected (#132, #206)output_contains_any expectation — New expectation field that passes when the agent response contains any one of the specified strings (#203)max_response_time_ms behavior rule — Enforce maximum response time constraints on agent execution (#201)prompt field can now reference an external file path instead of inline text (#157, #200)tool_calls grader — New grader type that validates the specific tool calls an agent makes during execution (#187, #202)run --output-dir now groups result files by timestamp for cleaner organization (#153)--discover finds eval.yaml in nested layout — Skill discovery now correctly locates eval.yaml files in evals/{name}/ directories at the project root (#44)Note truncated.
Nothing published for this version