What's Changed

• chore: Release v0.38.6 — registry and version sync by Shayne Boyer (@spboyer) in #527
• chore(deps): Bump Azure/setup-azd from 2.3.0 to 2.4.0 by Dependabot (@dependabot)[bot] in #536
• chore: expand default code owners by Shayne Boyer (@spboyer) in #541
• chore(deps): Bump dorny/paths-filter from 4.0.2 to 4.0.3 by Dependabot (@dependabot)[bot] in #528
• chore(deps): Bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.22.0 to 1.23.0 by Dependabot (@dependabot)[bot] in #529
• chore(deps-dev): Bump @playwright/test from 1.62.0 to 1.62.1 in /web by Dependabot (@dependabot)[bot] in #530
• chore(deps-dev): Bump eslint-plugin-react-refresh from 0.5.3 to 0.5.4 in /web by Dependabot (@dependabot)[bot] in #535
• chore(deps-dev): Bump @types/react from 19.2.17 to 19.2.18 in /web by Dependabot (@dependabot)[bot] in #539
• chore(deps-dev): Bump vite from 8.2.0 to 8.2.1 in /web by Dependabot (@dependabot)[bot] in #538
• chore(deps): Bump astro from 7.1.6 to 7.2.1 in /site by Dependabot (@dependabot)[bot] in #537
• chore(deps): Bump golang.org/x/text from 0.40.0 to 0.41.0 by Dependabot (@dependabot)[bot] in #534
• chore(deps): Bump @astrojs/starlight from 0.41.6 to 0.41.7 in /site by Dependabot (@dependabot)[bot] in #533
• chore(deps-dev): Bump globals from 17.9.0 to 17.11.0 in /web by Dependabot (@dependabot)[bot] in #532
• chore(deps): Bump github.com/azure/azure-dev/cli/azd from 1.29.0 to 1.31.1 by Dependabot (@dependabot)[bot] in #531

Full Changelog: v0.38.6...v0.38.7

Changelog

All notable changes to waza will be documented in this file.

The format is based on Keep a Changelog,
and this project adheres to Semantic Versioning.

Unreleased

0.38.6 - 2026-08-14

Added

• Remote grader references — graders[].ref can resolve Go-module-style GitHub references through waza get, with pinned lockfiles, digest verification, offline cache support, and runtime expansion (#15, #494).
• Registry CLI — Added waza registry search and waza registry add for discovering shared graders and adding compatible references to eval files (#17, #493).

Changed

• Dependencies — Updated OpenTelemetry, Azure Developer CLI, JSON Schema, Vite, React tooling, Playwright, Astro, Starlight, TypeScript ESLint, Lucide, and related lockfiles (#507-#521, #525).
• CI path filtering — Updated dorny/paths-filter to v4 for current runner support and merge-queue handling (#519).

0.38.5 - 2026-08-06

Fixed

• Tool-call argument grading — tool_calls graders now use canonical tool_events[].args data, preserving MCP and custom tool arguments across results.json round trips (#474, #477).
• Dev command output race — Trigger-discovery warnings are emitted before the Copilot spinner starts, preventing concurrent writes detected by the Go race suite (#506).

Changed

• Documentation site validation — Pull requests now validate the Starlight documentation site before merge (#473, #476).
• GitHub Actions hardening — Third-party actions are pinned to full commit SHAs and Dependabot applies a seven-day GitHub Actions cooldown (#504).
• Dependencies — Updated Playwright, ESLint, TypeScript ESLint, React Query, Starlight, Goldmark, and go-runewidth (#495, #497, #498, #499, #500, #502, #503).

0.38.4 - 2026-07-27

Fixed

• Release workflow idempotency — CLI artifact publishing now uploads assets to an existing release when the tag's release was created earlier in the workflow, instead of failing on gh release create.
• Windows update command — waza update now passes the PowerShell installer URL reliably, preventing Invoke-RestMethod from failing with a null or empty Uri (#448).
• JSON schema grader configuration — json_schema graders now accept the documented extract_json option in eval and task YAML, extracting exactly one JSON object or array candidate before schema validation (#457).
• Live MCP server tool availability — config.mcp_servers entries now default omitted tools to ["*"], matching MCP mock behavior so configured stdio and HTTP servers expose their tools to Copilot SDK evaluations (#449).

0.38.3 - 2026-07-17

Fixed

• MCP mock tool availability — mcp_mocks now explicitly exposes declared tools to the bundled Copilot CLI, preventing valid mock servers from being skipped during Copilot SDK evaluations (#440).

0.38.2 - 2026-07-16

Fixed

• JSON schema ambiguity detection — extract_json now rejects output containing a JSON code block plus any additional JSON document.

0.38.1 - 2026-07-16

Fixed

• JSON schema output extraction — The json_schema grader can now extract exactly one JSON document from prose or a Markdown JSON code block with extract_json: true, while retaining strict pure-JSON validation by default.
• APM-installed skill detection — Skill discovery recognizes skills installed by Azure API Management (#402).

Changed

• Dependencies and tooling — Updated the Copilot SDK, OpenTelemetry, azd, documentation tooling, dashboard dependencies, and GitHub Actions.
• Documentation — Added the executor to the getting-started manifest example (#435).

0.38.0 - 2026-06-30

Added

• Focused eval suggestions — waza suggest now supports targeted generation with --count, --focus, --dry-run, --apply, and --force (#357, #380)
• Per-turn checkpoint graders — Task YAML can run inline graders after specific conversation turns with checkpoints[] and on_failure policies (#358, #386)
• Rubric preset library — Prompt graders can reuse built-in rubric presets for common judge dimensions; no separate waza rubric subcommand ships in this release (#360, #381)
• Spec verification — Added waza spec verify to report eval coverage against SKILL.md requirements (#361, #385)
• OpenTelemetry trace export — Added waza run --otel-exporter, --otel-endpoint, --otel-headers, --otel-file, and --otel-include-payloads (#362, #383)
• MCP server mocks — Added eval-level mcp_mocks: for hermetic Copilot SDK tool-call evals (#363, #387)
• Regression gates — Added waza gate with stable exit codes for pass, regression, golden failure, and config errors (#364, #384)
• Adversarial harness — Added waza adversarial and eval-level adversarial: pack configuration for prompt-injection and scope-bypass checks (#365, #392)
• Tool metrics and structured argument matchers — Results now include normalized tool_events[]; tool graders can assert structured argument matchers through expect_tools[].args and tool_calls.expect[].args (#366, #388)
• Snapshot and replay — Added waza run --snapshot and waza replay with a self-contained snapshot artifact format (#367, #391)
• Schema version policy — Documented and enforced MAJOR.MINOR schemaVersion compatibility for public artifacts (#368, #382)
• Dashboard SSE resume — Added Last-Event-ID / lastEventId resume support for dashboard event streams, including legacy /api/events (#178, #397)

Changed

• Phase 1 internal refactor — Internal cleanup with no user-facing CLI, schema, or site behavior changes (#10)

0.37.0 - 2026-06-18

Added

• Interactive skill responder — Eval runs can now drive interactive skills with an LLM responder for more realistic conversational workflows (#304, closes #303)
• Triage automation and regression loop — Added triage automation and regression-loop support for Squad workflow validation (#326)

Fixed

• Task-level context fixtures — Task-level context fixtures now materialize in workspaces before execution (#329)
• waza suggest engine failures — Engine failures are now surfaced by waza suggest instead of being hidden behind success-shaped output (#330)
• Session idle hang — Upgraded the Copilot SDK to v1.0.2 and re-bundled Copilot CLI 1.0.64-0 to fix session idle hangs (#333)

Dependencies

• Bump astro from 6.3.2 to 6.4.7 in /site (#331)
• Bump js-yaml from 4.1.1 to 4.2.0 in /site (#327)

0.36.0 - 2026-06-15

Added

• Squad framework v0.10.0 upgrade — Upgraded Squad from 0.8.25 to 0.10.0 (#322, #323)
• Squad workflow failure detection — Added Squad workflow and failure detection infrastructure (#322, #324)

Fixed

• Prompt grader timeout configuration — Prompt grader timeout can now be configured with WAZA_PROMPT_GRADER_TIMEOUT (#319)
• Session-start hang detection — Added a first-event watchdog to catch session-start hangs (#321)
• Non-Squad coordinator canary handling — Clarified the Squad coordinator canary guard so non-Squad sessions can continue without using Squad (#325)

Dependencies

• Bump esbuild, @tailwindcss/vite, @vitejs/plugin-react, and vite (#317)

0.35.0 - 2026-06-06

Added

• Copilot SDK v1.0.0 upgrade — Upgraded github.com/github/copilot-sdk/go to v1.0.0 and surfaced premium-request credits on the dashboard (#311)
• Model-aware dashboard pricing — Dashboard cost calculation now applies per-model pricing for more accurate run cost reporting (#310)
• Git worktree resources in task inputs — Tasks can now reference git worktree resources as inputs (#121, #302)

Fixed

• BYOK + --model startup arg — The Copilot CLI validates the startup --model flag against the Copilot catalog before BYOK provider config is applied, so provider-only model IDs would fail. The --model startup arg is now skipped when a BYOK provider is configured (#305, #306)
• Model override propagation — --model is now passed via CLIArgs so it correctly overrides user settings and experiment flights (#263)
• Copilot CLI PATH fallback — Prevent silent fallback to a Copilot CLI on PATH when the bundled binary is unavailable (#300)
• Installer latest-release selection — Installer now correctly selects the latest standalone waza release (#299)
• Skill best practices doc link — Fixed the broken skill best practices reference (#295, #298)

Changed

• AgentEngine cancellation — Simplified AgentEngine cancellation handling around caller contexts to make shutdown semantics more predictable (#290)

0.34.0 - 2026-05-23

Added

• BYOK provider wiring — Added bring-your-own-key provider support for configured model providers (#240)
• waza update command — Added an update command for upgrading local Waza installations (#288)
• Skill injection opt-out — Added an option to run evals without injecting the target skill body (#285, #292)
• Forbidden skills grading — skill_invocation graders can now assert that specific skills must not be invoked (#286, #291)
• Per-trial usage reporting — Results JSON now includes per-trial usage details for deeper run analysis (#277)
• Agent-friendly GitHub templates — Added issue and pull request templates tuned for agent-authored work (#293)

Fixed

• Tool approval handling — Tool permission handling now uses the SDK approval kind (#240)
• Signal cancellation — waza run now respects cancellation signals more reliably (#279)
• Sandbox prompt handling — Empty sandbox prompts are guarded before execution (#273, #278)
• Custom agent example schema — Fixed the custom-agent eval example to match the supported schema (#282)
• Binary release links — Fixed binary release documentation links (#276, #284)
• Agent path guidance — Corrected AGENTS root path guidance (#267, #269)

Changed

• Run concurrency — waza run now reuses a shared Copilot client and auto-sizes parallel workers when --workers is unset (#135, #221)
• Documentation — Updated integration testing, custom-agent eval, and OpenAI Evals model-graded YAML documentation (#281, #283, #14, #280)
• Release workflow — GitHub Pages deployment now runs after the release workflow (#265)

0.33.0 - 2026-05-21

Note: This release includes the changes previously prepared under 0.32.0, which was not published.

Added

• Configurable eval file naming — .waza.yaml can now configure files.evalFile, files.taskGlob, and files.taskFileSuffix, with the new naming carried through scaffolding, workspace discovery, discovery mode, schemas, and docs while preserving the existing eval.yaml and tasks/*.yaml defaults (#254, closes #232)
• Instruction files in eval runs — Eval-level config.instruction_files and task-level instruction_files now copy files from the active context into task workspaces and append path-labeled contents to the Copilot system message (#248, closes #239)

Fixed

• Prompt graders use the execution engine — Prompt graders now route judge turns through CopilotEngine instead of constructing a Copilot client directly, keeping grader execution aligned with engine configuration and preserving follow-up recovery behavior (#258, closes #54)
• Prompt grader follow-up recovery — Prompt grading now preserves collected grades when a follow-up turn fails after successful grader collection (#251)
• Bundled Copilot CLI updated — Embedded copilot-cli bundles are updated from 1.0.2 to 1.0.49 across supported platforms, with reproducible pinned bundle generation via COPILOT_CLI_VERSION (#260, closes #244)
• Spec-aligned skill scaffolding — waza new skill no longer asks for a nonstandard skill type or emits type: frontmatter, and the wizard now rejects early exits that omit required name or description fields (#261, closes #243)
• waza check eval discovery — Nested skills and separated evals are discovered consistently in multi-skill workspaces (#247, closes #238)
• Skill body routing markers — Compliance scoring now detects trigger, anti-trigger, and routing markers in SKILL.md body sections as well as frontmatter descriptions (#236, closes #223)

Changed

• Copilot SDK v0.3.0 migration — Updated github.com/github/copilot-sdk/go to v0.3.0, migrated session event handling to typed payloads, and refreshed transcript, logging, web API, usage collection, suggestion trace, and test coverage for the new API (#255, closes #253)
• Dashboard validation coverage — Added coverage for dashboard lint and end-to-end validation (#249)
• Install documentation — Replaced unsupported go install guidance and clarified Windows/WSL install behavior (#246, closes #242; #245, closes #241)
• Dependencies — Bump devalue in /site, postcss in /web, and astro in /site (#237, #235, #234)

0.31.0 - 2026-04-28

Added

• Custom agent (.agent.md) eval support — Discover .agent.md files alongside SKILL.md, parse agent-specific frontmatter (tools, model, handoffs, mcp-servers, agents), auto-inject tool_constraint grader from agent tools: field, complete worked example under examples/custom-agent/, and new "Evaluating Custom Agents" docs guide (#226, closes #225)

Fixed

• Mock engine echoes file content — _output_contains expectations against file contents now work in CI without a real model. Mock response includes task metadata, file paths, and a 1KB content preview per resource (#228, closes #227)
• waza serve no longer crashes when stdin isn't a terminal — MCP stdio server only starts when term.IsTerminal() is true; piped input or background mode no longer kills the HTTP dashboard (#224)

Changed

• Vocabulary renames — Internal types renamed: BenchmarkSpec → EvalSpec, TestRunner → EvalRunner. Not a breaking change for external consumers (types live in internal/) (#222)

Documentation

• Cross-reference audit for recent renames + custom agent feature: added .agent.md coverage to quickstart, getting-started, GUIDE, TUTORIAL, examples README; updated mock engine descriptions in INTEGRATION-TESTING and eval-yaml guide (#230)

Dependencies

• Bump postcss from 8.5.6 to 8.5.12 in /site (#229)

0.30.1 - 2026-04-22

Documentation

• Updated README with missing CLI commands — Added documentation for recently-added CLI commands that were missing from the README (#220)

0.30.0 - 2026-04-22

Added

• waza quality command — LLM-as-Judge skill quality scoring that evaluates skill output quality using a configurable judge model (#218)
• Scope-reduction advisory check — waza check now includes an advisory that flags skills with overly broad scope, helping authors tighten skill definitions (#219)

0.29.0 - 2026-04-22

Added

• --keep-workspace flag — Preserve the temporary workspace after task execution for debugging agent output (#123, #217)
• --no-skills flag and disabled_skills config — Disable specific skills during evaluation to isolate behavior (#126, #216)
• Non-blocking version update check — CLI now checks for newer waza versions in the background without slowing startup (#104, #214)
• Per-task skill_directories — Specify different skill directories for individual tasks in eval YAML (#156, #215)

Dependencies

• Bump astro and @astrojs/starlight in /site (#212)

0.28.0 - 2026-04-21

Added

• Follow-up prompts in eval YAML — Tasks can now include pre-written follow-up prompts for multi-turn evaluation conversations (#189, #209)
• waza models command — List all available models supported by the configured engine (#208)
• Early termination for trigger tests — Trigger tests can now stop early once the target skill is invoked, reducing evaluation time (#207)

Fixed

• Stricter YAML validation — Audited all YAML parsers; unknown fields in TestCase definitions are now properly rejected (#132, #206)
• Test fixture assertion syntax — Fixed invalid Python expression in a test fixture assertion (#197)
• CI integration test stability — CI integration tests now correctly handle expected eval failures when using the mock executor (#210)

Documentation

• Added Quick Start guide to the documentation site (#205)

0.27.0 - 2026-04-21

Added

• output_contains_any expectation — New expectation field that passes when the agent response contains any one of the specified strings (#203)
• max_response_time_ms behavior rule — Enforce maximum response time constraints on agent execution (#201)
• Task prompt from file — Task prompt field can now reference an external file path instead of inline text (#157, #200)
• tool_calls grader — New grader type that validates the specific tool calls an agent makes during execution (#187, #202)

Fixed

• Webserver test resilience — Webserver tests now skip gracefully when frontend assets are not built (#204)

0.26.0 - 2026-04-21

Changed

• Timestamped output directories — run --output-dir now groups result files by timestamp for cleaner organization (#153)
• Improved debug logging — Debug output is now more structured and useful for troubleshooting (#152)

Fixed

• --discover finds eval.yaml in nested layout — Skill discovery now correctly locates eval.yaml files in evals/{name}/ directories at the project root (#44)
• Diff grader reads post-execution workspace — The diff grader now reads files fr

Note truncated.

Waza v0.38.7

Compare

Choose a tag to compare

What's Changed

• chore: update uncovered dependencies by Shayne Boyer (@spboyer) in #525
• chore(deps): Bump js-yaml from 4.3.0 to 4.3.1 in /site by Dependabot (@dependabot)[bot] in #507
• chore(deps): Bump github.com/santhosh-tekuri/jsonschema/v6 from 6.0.2 to 6.0.3 by Dependabot (@dependabot)[bot] in #510
• chore(deps): Bump astro from 7.1.4 to 7.1.6 in /site by Dependabot (@dependabot)[bot] in #512
• chore(deps): Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp from 1.44.0 to 1.45.0 by Dependabot (@dependabot)[bot] in #508
• chore(deps): Bump dorny/paths-filter from 3.0.4 to 4.0.2 by Dependabot (@dependabot)[bot] in #519
• chore(deps): Bump github.com/azure/azure-dev/cli/azd from 1.28.1 to 1.29.0 by Dependabot (@dependabot)[bot] in #516
• chore(deps-dev): Bump @playwright/test from 1.62.0 to 1.62.1 in /site by Dependabot (@dependabot)[bot] in #517
• chore(deps-dev): Bump vite from 8.1.5 to 8.2.0 in /web by Dependabot (@dependabot)[bot] in #518
• chore(deps-dev): Bump @vitejs/plugin-react from 6.0.3 to 6.0.5 in /web by Dependabot (@dependabot)[bot] in #520
• chore(deps): Bump lucide-react from 1.24.0 to 1.28.0 in /web by Dependabot (@dependabot)[bot] in #513
• chore(deps): Bump go.opentelemetry.io/otel/exporters/stdout/stdouttrace from 1.44.0 to 1.45.0 by Dependabot (@dependabot)[bot] in #509
• chore(deps-dev): Bump globals from 17.7.0 to 17.9.0 in /web by Dependabot (@dependabot)[bot] in #515
• chore(deps): Bump @astrojs/starlight from 0.41.5 to 0.41.6 in /site by Dependabot (@dependabot)[bot] in #514
• chore(deps-dev): Bump typescript-eslint from 8.65.0 to 8.66.0 in /web by Dependabot (@dependabot)[bot] in #521
• feat: add remote grader refs by Shayne Boyer (@spboyer) in #494
• feat: add registry CLI search and add by Shayne Boyer (@spboyer) in #493

Full Changelog: v0.38.5...v0.38.6

Changelog

All notable changes to waza will be documented in this file.

The format is based on Keep a Changelog,
and this project adheres to Semantic Versioning.

Unreleased

0.38.5 - 2026-08-06

Fixed

• Tool-call argument grading — tool_calls graders now use canonical tool_events[].args data, preserving MCP and custom tool arguments across results.json round trips (#474, #477).
• Dev command output race — Trigger-discovery warnings are emitted before the Copilot spinner starts, preventing concurrent writes detected by the Go race suite (#506).

Changed

• Documentation site validation — Pull requests now validate the Starlight documentation site before merge (#473, #476).
• GitHub Actions hardening — Third-party actions are pinned to full commit SHAs and Dependabot applies a seven-day GitHub Actions cooldown (#504).
• Dependencies — Updated Playwright, ESLint, TypeScript ESLint, React Query, Starlight, Goldmark, and go-runewidth (#495, #497, #498, #499, #500, #502, #503).

0.38.4 - 2026-07-27

Fixed

• Release workflow idempotency — CLI artifact publishing now uploads assets to an existing release when the tag's release was created earlier in the workflow, instead of failing on gh release create.
• Windows update command — waza update now passes the PowerShell installer URL reliably, preventing Invoke-RestMethod from failing with a null or empty Uri (#448).
• JSON schema grader configuration — json_schema graders now accept the documented extract_json option in eval and task YAML, extracting exactly one JSON object or array candidate before schema validation (#457).
• Live MCP server tool availability — config.mcp_servers entries now default omitted tools to ["*"], matching MCP mock behavior so configured stdio and HTTP servers expose their tools to Copilot SDK evaluations (#449).

0.38.3 - 2026-07-17

Fixed

• MCP mock tool availability — mcp_mocks now explicitly exposes declared tools to the bundled Copilot CLI, preventing valid mock servers from being skipped during Copilot SDK evaluations (#440).

0.38.2 - 2026-07-16

Fixed

• JSON schema ambiguity detection — extract_json now rejects output containing a JSON code block plus any additional JSON document.

0.38.1 - 2026-07-16

Fixed

• JSON schema output extraction — The json_schema grader can now extract exactly one JSON document from prose or a Markdown JSON code block with extract_json: true, while retaining strict pure-JSON validation by default.
• APM-installed skill detection — Skill discovery recognizes skills installed by Azure API Management (#402).

Changed

• Dependencies and tooling — Updated the Copilot SDK, OpenTelemetry, azd, documentation tooling, dashboard dependencies, and GitHub Actions.
• Documentation — Added the executor to the getting-started manifest example (#435).

0.38.0 - 2026-06-30

Added

• Focused eval suggestions — waza suggest now supports targeted generation with --count, --focus, --dry-run, --apply, and --force (#357, #380)
• Per-turn checkpoint graders — Task YAML can run inline graders after specific conversation turns with checkpoints[] and on_failure policies (#358, #386)
• Rubric preset library — Prompt graders can reuse built-in rubric presets for common judge dimensions; no separate waza rubric subcommand ships in this release (#360, #381)
• Spec verification — Added waza spec verify to report eval coverage against SKILL.md requirements (#361, #385)
• OpenTelemetry trace export — Added waza run --otel-exporter, --otel-endpoint, --otel-headers, --otel-file, and --otel-include-payloads (#362, #383)
• MCP server mocks — Added eval-level mcp_mocks: for hermetic Copilot SDK tool-call evals (#363, #387)
• Regression gates — Added waza gate with stable exit codes for pass, regression, golden failure, and config errors (#364, #384)
• Adversarial harness — Added waza adversarial and eval-level adversarial: pack configuration for prompt-injection and scope-bypass checks (#365, #392)
• Tool metrics and structured argument matchers — Results now include normalized tool_events[]; tool graders can assert structured argument matchers through expect_tools[].args and tool_calls.expect[].args (#366, #388)
• Snapshot and replay — Added waza run --snapshot and waza replay with a self-contained snapshot artifact format (#367, #391)
• Schema version policy — Documented and enforced MAJOR.MINOR schemaVersion compatibility for public artifacts (#368, #382)
• Dashboard SSE resume — Added Last-Event-ID / lastEventId resume support for dashboard event streams, including legacy /api/events (#178, #397)

Changed

• Phase 1 internal refactor — Internal cleanup with no user-facing CLI, schema, or site behavior changes (#10)

0.37.0 - 2026-06-18

Added

• Interactive skill responder — Eval runs can now drive interactive skills with an LLM responder for more realistic conversational workflows (#304, closes #303)
• Triage automation and regression loop — Added triage automation and regression-loop support for Squad workflow validation (#326)

Fixed

• Task-level context fixtures — Task-level context fixtures now materialize in workspaces before execution (#329)
• waza suggest engine failures — Engine failures are now surfaced by waza suggest instead of being hidden behind success-shaped output (#330)
• Session idle hang — Upgraded the Copilot SDK to v1.0.2 and re-bundled Copilot CLI 1.0.64-0 to fix session idle hangs (#333)

Dependencies

• Bump astro from 6.3.2 to 6.4.7 in /site (#331)
• Bump js-yaml from 4.1.1 to 4.2.0 in /site (#327)

0.36.0 - 2026-06-15

Added

• Squad framework v0.10.0 upgrade — Upgraded Squad from 0.8.25 to 0.10.0 (#322, #323)
• Squad workflow failure detection — Added Squad workflow and failure detection infrastructure (#322, #324)

Fixed

• Prompt grader timeout configuration — Prompt grader timeout can now be configured with WAZA_PROMPT_GRADER_TIMEOUT (#319)
• Session-start hang detection — Added a first-event watchdog to catch session-start hangs (#321)
• Non-Squad coordinator canary handling — Clarified the Squad coordinator canary guard so non-Squad sessions can continue without using Squad (#325)

Dependencies

• Bump esbuild, @tailwindcss/vite, @vitejs/plugin-react, and vite (#317)

0.35.0 - 2026-06-06

Added

• Copilot SDK v1.0.0 upgrade — Upgraded github.com/github/copilot-sdk/go to v1.0.0 and surfaced premium-request credits on the dashboard (#311)
• Model-aware dashboard pricing — Dashboard cost calculation now applies per-model pricing for more accurate run cost reporting (#310)
• Git worktree resources in task inputs — Tasks can now reference git worktree resources as inputs (#121, #302)

Fixed

• BYOK + --model startup arg — The Copilot CLI validates the startup --model flag against the Copilot catalog before BYOK provider config is applied, so provider-only model IDs would fail. The --model startup arg is now skipped when a BYOK provider is configured (#305, #306)
• Model override propagation — --model is now passed via CLIArgs so it correctly overrides user settings and experiment flights (#263)
• Copilot CLI PATH fallback — Prevent silent fallback to a Copilot CLI on PATH when the bundled binary is unavailable (#300)
• Installer latest-release selection — Installer now correctly selects the latest standalone waza release (#299)
• Skill best practices doc link — Fixed the broken skill best practices reference (#295, #298)

Changed

• AgentEngine cancellation — Simplified AgentEngine cancellation handling around caller contexts to make shutdown semantics more predictable (#290)

0.34.0 - 2026-05-23

Added

• BYOK provider wiring — Added bring-your-own-key provider support for configured model providers (#240)
• waza update command — Added an update command for upgrading local Waza installations (#288)
• Skill injection opt-out — Added an option to run evals without injecting the target skill body (#285, #292)
• Forbidden skills grading — skill_invocation graders can now assert that specific skills must not be invoked (#286, #291)
• Per-trial usage reporting — Results JSON now includes per-trial usage details for deeper run analysis (#277)
• Agent-friendly GitHub templates — Added issue and pull request templates tuned for agent-authored work (#293)

Fixed

• Tool approval handling — Tool permission handling now uses the SDK approval kind (#240)
• Signal cancellation — waza run now respects cancellation signals more reliably (#279)
• Sandbox prompt handling — Empty sandbox prompts are guarded before execution (#273, #278)
• Custom agent example schema — Fixed the custom-agent eval example to match the supported schema (#282)
• Binary release links — Fixed binary release documentation links (#276, #284)
• Agent path guidance — Corrected AGENTS root path guidance (#267, #269)

Changed

• Run concurrency — waza run now reuses a shared Copilot client and auto-sizes parallel workers when --workers is unset (#135, #221)
• Documentation — Updated integration testing, custom-agent eval, and OpenAI Evals model-graded YAML documentation (#281, #283, #14, #280)
• Release workflow — GitHub Pages deployment now runs after the release workflow (#265)

0.33.0 - 2026-05-21

Note: This release includes the changes previously prepared under 0.32.0, which was not published.

Added

• Configurable eval file naming — .waza.yaml can now configure files.evalFile, files.taskGlob, and files.taskFileSuffix, with the new naming carried through scaffolding, workspace discovery, discovery mode, schemas, and docs while preserving the existing eval.yaml and tasks/*.yaml defaults (#254, closes #232)
• Instruction files in eval runs — Eval-level config.instruction_files and task-level instruction_files now copy files from the active context into task workspaces and append path-labeled contents to the Copilot system message (#248, closes #239)

Fixed

• Prompt graders use the execution engine — Prompt graders now route judge turns through CopilotEngine instead of constructing a Copilot client directly, keeping grader execution aligned with engine configuration and preserving follow-up recovery behavior (#258, closes #54)
• Prompt grader follow-up recovery — Prompt grading now preserves collected grades when a follow-up turn fails after successful grader collection (#251)
• Bundled Copilot CLI updated — Embedded copilot-cli bundles are updated from 1.0.2 to 1.0.49 across supported platforms, with reproducible pinned bundle generation via COPILOT_CLI_VERSION (#260, closes #244)
• Spec-aligned skill scaffolding — waza new skill no longer asks for a nonstandard skill type or emits type: frontmatter, and the wizard now rejects early exits that omit required name or description fields (#261, closes #243)
• waza check eval discovery — Nested skills and separated evals are discovered consistently in multi-skill workspaces (#247, closes #238)
• Skill body routing markers — Compliance scoring now detects trigger, anti-trigger, and routing markers in SKILL.md body sections as well as frontmatter descriptions (#236, closes #223)

Changed

• Copilot SDK v0.3.0 migration — Updated github.com/github/copilot-sdk/go to v0.3.0, migrated session event handling to typed payloads, and refreshed transcript, logging, web API, usage collection, suggestion trace, and test coverage for the new API (#255, closes #253)
• Dashboard validation coverage — Added coverage for dashboard lint and end-to-end validation (#249)
• Install documentation — Replaced unsupported go install guidance and clarified Windows/WSL install behavior (#246, closes #242; #245, closes #241)
• Dependencies — Bump devalue in /site, postcss in /web, and astro in /site (#237, #235, #234)

0.31.0 - 2026-04-28

Added

• Custom agent (.agent.md) eval support — Discover .agent.md files alongside SKILL.md, parse agent-specific frontmatter (tools, model, handoffs, mcp-servers, agents), auto-inject tool_constraint grader from agent tools: field, complete worked example under examples/custom-agent/, and new "Evaluating Custom Agents" docs guide (#226, closes #225)

Fixed

• Mock engine echoes file content — _output_contains expectations against file contents now work in CI without a real model. Mock response includes task metadata, file paths, and a 1KB content preview per resource (#228, closes #227)
• waza serve no longer crashes when stdin isn't a terminal — MCP stdio server only starts when term.IsTerminal() is true; piped input or background mode no longer kills the HTTP dashboard (#224)

Changed

• Vocabulary renames — Internal types renamed: BenchmarkSpec → EvalSpec, TestRunner → EvalRunner. Not a breaking change for external consumers (types live in internal/) (#222)

Documentation

• Cross-reference audit for recent renames + custom agent feature: added .agent.md coverage to quickstart, getting-started, GUIDE, TUTORIAL, examples README; updated mock engine descriptions in INTEGRATION-TESTING and eval-yaml guide (#230)

Dependencies

• Bump postcss from 8.5.6 to 8.5.12 in /site (#229)

0.30.1 - 2026-04-22

Documentation

• Updated README with missing CLI commands — Added documentation for recently-added CLI commands that were missing from the README (#220)

0.30.0 - 2026-04-22

Added

• waza quality command — LLM-as-Judge skill quality scoring that evaluates skill output quality using a configurable judge model (#218)
• Scope-reduction advisory check — waza check now includes an advisory that flags skills with overly broad scope, helping authors tighten skill definitions (#219)

0.29.0 - 2026-04-22

Added

• --keep-workspace flag — Preserve the temporary workspace after task execution for debugging agent output (#123, #217)
• --no-skills flag and disabled_skills config — Disable specific skills during evaluation to isolate behavior (#126, #216)
• Non-blocking version update check — CLI now checks for newer waza versions in the background without slowing startup (#104, #214)
• Per-task skill_directories — Specify different skill directories for individual tasks in eval YAML (#156, #215)

Dependencies

• Bump astro and @astrojs/starlight in /site (#212)

0.28.0 - 2026-04-21

Added

• Follow-up prompts in eval YAML — Tasks can now include pre-written follow-up prompts for multi-turn evaluation conversations (#189, #209)
• waza models command — List all available models supported by the configured engine (#208)
• Early termination for trigger tests — Trigger tests can now stop early once the target skill is invoked, reducing evaluation time (#207)

Fixed

• Stricter YAML validation — Audited all YAML parsers; unknown fields in TestCase definitions are now properly rejected (#132, #206)
• Test fixture assertion syntax — Fixed invalid Python expression in a test fixture assertion (#197)
• CI integration test stability — CI integration tests now correctly handle expected eval failures when using the mock executor (#210)

Documentation

• Added Quick Start guide to the documentation site (#205)

0.27.0 - 2026-04-21

Added

• output_contains_any expectation — New expectation field that passes when the agent response contains any one of the specified strings (#203)
• max_response_time_ms behavior rule — Enforce maximum response time constraints on agent execution (#201)
• Task prompt from file — Task prompt field can now reference an external file path instead of inline text (#157, #200)
• tool_calls grader — New grader type that validates the specific tool calls an agent makes during execution (#187, #202)

Fixed

• Webserver test resilience — Webserver tests now skip gracefully when frontend assets are not built (#204)

0.26.0 - 2026-04-21

Changed

• Timestamped output directories — run --output-dir now groups result files by timestamp for cleaner organization (#153)
• Improved debug logging — Debug output is now more structured and useful for troubleshooting (#152)

Fixed

• --discover finds eval.yaml in nested layout — Skill discovery now correctly locates eval.yaml files in evals/{name}/ directories at the project root (#44)
• Diff grader reads post-execution workspace — The diff grader now reads files from the workspace after agent execution completes, not before (#165, #196)
• Grader config validation — Required grader configuration fields are now validated before evaluation starts (#195)
• macOS install and trigger test count — Fixed macOS binary installation and an off-by-one error in trigger test counting (#164, #184, #193)

Documentation

• Added cache command reference, prompt mode documentation, and complete YAML schema reference (#198)
• Updated demo guide and added CI/CD integration guide (#112, #89,

Note truncated.

Added

• Remote grader references — graders[].ref can resolve Go-module-style GitHub references through waza get, with pinned lockfiles, digest verification, offline cache support, and runtime expansion (#15, #494).
• Registry CLI — Added waza registry search and waza registry add for discovering shared graders and adding compatible references to eval files (#17, #493).

Changed

• Dependencies — Updated OpenTelemetry, Azure Developer CLI, JSON Schema, Vite, React tooling, Playwright, Astro, Starlight, TypeScript ESLint, Lucide, and related lockfiles (#507-#521, #525).
• CI path filtering — Updated dorny/paths-filter to v4 for current runner support and merge-queue handling (#519).

Added

• Remote grader references — graders[].ref can resolve Go-module-style GitHub references through waza get , with pinned lockfiles, digest verification, offline cache support, and runtime expansion ( #15 , #494 ).

• Registry CLI — Added waza registry search and waza registry add for discovering shared graders and adding compatible references to eval files ( #17 , #493 ).

Changed

• Dependencies — Updated OpenTelemetry, Azure Developer CLI, JSON Schema, Vite, React tooling, Playwright, Astro, Starlight, TypeScript ESLint, Lucide, and related lockfiles ( #507 - #521 , #525 ).

• CI path filtering — Updated dorny/paths-filter to v4 for current runner support and merge-queue handling ( #519 ).

What's Changed

• chore(deps-dev): Bump eslint from 10.7.0 to 10.8.0 in /web by Dependabot (@dependabot)[bot] in #502
• chore(deps): Bump @astrojs/starlight from 0.41.4 to 0.41.5 in /site by Dependabot (@dependabot)[bot] in #500
• chore(deps): Bump github.com/yuin/goldmark from 1.8.4 to 1.8.5 by Dependabot (@dependabot)[bot] in #499
• chore(deps): Bump github.com/mattn/go-runewidth from 0.0.24 to 0.0.27 by Dependabot (@dependabot)[bot] in #498
• chore(deps): Bump @tanstack/react-query from 5.101.2 to 5.101.4 in /web by Dependabot (@dependabot)[bot] in #497
• chore(deps-dev): Bump typescript-eslint from 8.63.0 to 8.65.0 in /web by Dependabot (@dependabot)[bot] in #495
• Fix tool_calls args grading from tool_events by Shayne Boyer (@spboyer) in #477
• Pin GitHub Actions to full-length commit SHAs by Microsoft Open Source Security Bot (@OssSecurityBot) in #504
• chore(deps-dev): Bump @playwright/test from 1.61.1 to 1.62.0 in /web by Dependabot (@dependabot)[bot] in #503
• ci: validate site builds on pull requests by Shayne Boyer (@spboyer) in #476
• fix: avoid concurrent spinner output by Shayne Boyer (@spboyer) in #506
• Release v0.38.5 by Shayne Boyer (@spboyer) in #522

New Contributors

• Microsoft Open Source Security Bot (@OssSecurityBot) made their first contribution in #504

Full Changelog: v0.38.4...v0.38.5

Changelog

All notable changes to waza will be documented in this file.

The format is based on Keep a Changelog,
and this project adheres to Semantic Versioning.

Unreleased

0.38.5 - 2026-08-06

Fixed

• Tool-call argument grading — tool_calls graders now use canonical tool_events[].args data, preserving MCP and custom tool arguments across results.json round trips (#474, #477).
• Dev command output race — Trigger-discovery warnings are emitted before the Copilot spinner starts, preventing concurrent writes detected by the Go race suite (#506).

Changed

• Documentation site validation — Pull requests now validate the Starlight documentation site before merge (#473, #476).
• GitHub Actions hardening — Third-party actions are pinned to full commit SHAs and Dependabot applies a seven-day GitHub Actions cooldown (#504).
• Dependencies — Updated Playwright, ESLint, TypeScript ESLint, React Query, Starlight, Goldmark, and go-runewidth (#495, #497, #498, #499, #500, #502, #503).

0.38.4 - 2026-07-27

Fixed

• Release workflow idempotency — CLI artifact publishing now uploads assets to an existing release when the tag's release was created earlier in the workflow, instead of failing on gh release create.
• Windows update command — waza update now passes the PowerShell installer URL reliably, preventing Invoke-RestMethod from failing with a null or empty Uri (#448).
• JSON schema grader configuration — json_schema graders now accept the documented extract_json option in eval and task YAML, extracting exactly one JSON object or array candidate before schema validation (#457).
• Live MCP server tool availability — config.mcp_servers entries now default omitted tools to ["*"], matching MCP mock behavior so configured stdio and HTTP servers expose their tools to Copilot SDK evaluations (#449).

0.38.3 - 2026-07-17

Fixed

• MCP mock tool availability — mcp_mocks now explicitly exposes declared tools to the bundled Copilot CLI, preventing valid mock servers from being skipped during Copilot SDK evaluations (#440).

0.38.2 - 2026-07-16

Fixed

• JSON schema ambiguity detection — extract_json now rejects output containing a JSON code block plus any additional JSON document.

0.38.1 - 2026-07-16

Fixed

• JSON schema output extraction — The json_schema grader can now extract exactly one JSON document from prose or a Markdown JSON code block with extract_json: true, while retaining strict pure-JSON validation by default.
• APM-installed skill detection — Skill discovery recognizes skills installed by Azure API Management (#402).

Changed

• Dependencies and tooling — Updated the Copilot SDK, OpenTelemetry, azd, documentation tooling, dashboard dependencies, and GitHub Actions.
• Documentation — Added the executor to the getting-started manifest example (#435).

0.38.0 - 2026-06-30

Added

• Focused eval suggestions — waza suggest now supports targeted generation with --count, --focus, --dry-run, --apply, and --force (#357, #380)
• Per-turn checkpoint graders — Task YAML can run inline graders after specific conversation turns with checkpoints[] and on_failure policies (#358, #386)
• Rubric preset library — Prompt graders can reuse built-in rubric presets for common judge dimensions; no separate waza rubric subcommand ships in this release (#360, #381)
• Spec verification — Added waza spec verify to report eval coverage against SKILL.md requirements (#361, #385)
• OpenTelemetry trace export — Added waza run --otel-exporter, --otel-endpoint, --otel-headers, --otel-file, and --otel-include-payloads (#362, #383)
• MCP server mocks — Added eval-level mcp_mocks: for hermetic Copilot SDK tool-call evals (#363, #387)
• Regression gates — Added waza gate with stable exit codes for pass, regression, golden failure, and config errors (#364, #384)
• Adversarial harness — Added waza adversarial and eval-level adversarial: pack configuration for prompt-injection and scope-bypass checks (#365, #392)
• Tool metrics and structured argument matchers — Results now include normalized tool_events[]; tool graders can assert structured argument matchers through expect_tools[].args and tool_calls.expect[].args (#366, #388)
• Snapshot and replay — Added waza run --snapshot and waza replay with a self-contained snapshot artifact format (#367, #391)
• Schema version policy — Documented and enforced MAJOR.MINOR schemaVersion compatibility for public artifacts (#368, #382)
• Dashboard SSE resume — Added Last-Event-ID / lastEventId resume support for dashboard event streams, including legacy /api/events (#178, #397)

Changed

• Phase 1 internal refactor — Internal cleanup with no user-facing CLI, schema, or site behavior changes (#10)

0.37.0 - 2026-06-18

Added

• Interactive skill responder — Eval runs can now drive interactive skills with an LLM responder for more realistic conversational workflows (#304, closes #303)
• Triage automation and regression loop — Added triage automation and regression-loop support for Squad workflow validation (#326)

Fixed

• Task-level context fixtures — Task-level context fixtures now materialize in workspaces before execution (#329)
• waza suggest engine failures — Engine failures are now surfaced by waza suggest instead of being hidden behind success-shaped output (#330)
• Session idle hang — Upgraded the Copilot SDK to v1.0.2 and re-bundled Copilot CLI 1.0.64-0 to fix session idle hangs (#333)

Dependencies

• Bump astro from 6.3.2 to 6.4.7 in /site (#331)
• Bump js-yaml from 4.1.1 to 4.2.0 in /site (#327)

0.36.0 - 2026-06-15

Added

• Squad framework v0.10.0 upgrade — Upgraded Squad from 0.8.25 to 0.10.0 (#322, #323)
• Squad workflow failure detection — Added Squad workflow and failure detection infrastructure (#322, #324)

Fixed

• Prompt grader timeout configuration — Prompt grader timeout can now be configured with WAZA_PROMPT_GRADER_TIMEOUT (#319)
• Session-start hang detection — Added a first-event watchdog to catch session-start hangs (#321)
• Non-Squad coordinator canary handling — Clarified the Squad coordinator canary guard so non-Squad sessions can continue without using Squad (#325)

Dependencies

• Bump esbuild, @tailwindcss/vite, @vitejs/plugin-react, and vite (#317)

0.35.0 - 2026-06-06

Added

• Copilot SDK v1.0.0 upgrade — Upgraded github.com/github/copilot-sdk/go to v1.0.0 and surfaced premium-request credits on the dashboard (#311)
• Model-aware dashboard pricing — Dashboard cost calculation now applies per-model pricing for more accurate run cost reporting (#310)
• Git worktree resources in task inputs — Tasks can now reference git worktree resources as inputs (#121, #302)

Fixed

• BYOK + --model startup arg — The Copilot CLI validates the startup --model flag against the Copilot catalog before BYOK provider config is applied, so provider-only model IDs would fail. The --model startup arg is now skipped when a BYOK provider is configured (#305, #306)
• Model override propagation — --model is now passed via CLIArgs so it correctly overrides user settings and experiment flights (#263)
• Copilot CLI PATH fallback — Prevent silent fallback to a Copilot CLI on PATH when the bundled binary is unavailable (#300)
• Installer latest-release selection — Installer now correctly selects the latest standalone waza release (#299)
• Skill best practices doc link — Fixed the broken skill best practices reference (#295, #298)

Changed

• AgentEngine cancellation — Simplified AgentEngine cancellation handling around caller contexts to make shutdown semantics more predictable (#290)

0.34.0 - 2026-05-23

Added

• BYOK provider wiring — Added bring-your-own-key provider support for configured model providers (#240)
• waza update command — Added an update command for upgrading local Waza installations (#288)
• Skill injection opt-out — Added an option to run evals without injecting the target skill body (#285, #292)
• Forbidden skills grading — skill_invocation graders can now assert that specific skills must not be invoked (#286, #291)
• Per-trial usage reporting — Results JSON now includes per-trial usage details for deeper run analysis (#277)
• Agent-friendly GitHub templates — Added issue and pull request templates tuned for agent-authored work (#293)

Fixed

• Tool approval handling — Tool permission handling now uses the SDK approval kind (#240)
• Signal cancellation — waza run now respects cancellation signals more reliably (#279)
• Sandbox prompt handling — Empty sandbox prompts are guarded before execution (#273, #278)
• Custom agent example schema — Fixed the custom-agent eval example to match the supported schema (#282)
• Binary release links — Fixed binary release documentation links (#276, #284)
• Agent path guidance — Corrected AGENTS root path guidance (#267, #269)

Changed

• Run concurrency — waza run now reuses a shared Copilot client and auto-sizes parallel workers when --workers is unset (#135, #221)
• Documentation — Updated integration testing, custom-agent eval, and OpenAI Evals model-graded YAML documentation (#281, #283, #14, #280)
• Release workflow — GitHub Pages deployment now runs after the release workflow (#265)

0.33.0 - 2026-05-21

Note: This release includes the changes previously prepared under 0.32.0, which was not published.

Added

• Configurable eval file naming — .waza.yaml can now configure files.evalFile, files.taskGlob, and files.taskFileSuffix, with the new naming carried through scaffolding, workspace discovery, discovery mode, schemas, and docs while preserving the existing eval.yaml and tasks/*.yaml defaults (#254, closes #232)
• Instruction files in eval runs — Eval-level config.instruction_files and task-level instruction_files now copy files from the active context into task workspaces and append path-labeled contents to the Copilot system message (#248, closes #239)

Fixed

• Prompt graders use the execution engine — Prompt graders now route judge turns through CopilotEngine instead of constructing a Copilot client directly, keeping grader execution aligned with engine configuration and preserving follow-up recovery behavior (#258, closes #54)
• Prompt grader follow-up recovery — Prompt grading now preserves collected grades when a follow-up turn fails after successful grader collection (#251)
• Bundled Copilot CLI updated — Embedded copilot-cli bundles are updated from 1.0.2 to 1.0.49 across supported platforms, with reproducible pinned bundle generation via COPILOT_CLI_VERSION (#260, closes #244)
• Spec-aligned skill scaffolding — waza new skill no longer asks for a nonstandard skill type or emits type: frontmatter, and the wizard now rejects early exits that omit required name or description fields (#261, closes #243)
• waza check eval discovery — Nested skills and separated evals are discovered consistently in multi-skill workspaces (#247, closes #238)
• Skill body routing markers — Compliance scoring now detects trigger, anti-trigger, and routing markers in SKILL.md body sections as well as frontmatter descriptions (#236, closes #223)

Changed

• Copilot SDK v0.3.0 migration — Updated github.com/github/copilot-sdk/go to v0.3.0, migrated session event handling to typed payloads, and refreshed transcript, logging, web API, usage collection, suggestion trace, and test coverage for the new API (#255, closes #253)
• Dashboard validation coverage — Added coverage for dashboard lint and end-to-end validation (#249)
• Install documentation — Replaced unsupported go install guidance and clarified Windows/WSL install behavior (#246, closes #242; #245, closes #241)
• Dependencies — Bump devalue in /site, postcss in /web, and astro in /site (#237, #235, #234)

0.31.0 - 2026-04-28

Added

• Custom agent (.agent.md) eval support — Discover .agent.md files alongside SKILL.md, parse agent-specific frontmatter (tools, model, handoffs, mcp-servers, agents), auto-inject tool_constraint grader from agent tools: field, complete worked example under examples/custom-agent/, and new "Evaluating Custom Agents" docs guide (#226, closes #225)

Fixed

• Mock engine echoes file content — _output_contains expectations against file contents now work in CI without a real model. Mock response includes task metadata, file paths, and a 1KB content preview per resource (#228, closes #227)
• waza serve no longer crashes when stdin isn't a terminal — MCP stdio server only starts when term.IsTerminal() is true; piped input or background mode no longer kills the HTTP dashboard (#224)

Changed

• Vocabulary renames — Internal types renamed: BenchmarkSpec → EvalSpec, TestRunner → EvalRunner. Not a breaking change for external consumers (types live in internal/) (#222)

Documentation

• Cross-reference audit for recent renames + custom agent feature: added .agent.md coverage to quickstart, getting-started, GUIDE, TUTORIAL, examples README; updated mock engine descriptions in INTEGRATION-TESTING and eval-yaml guide (#230)

Dependencies

• Bump postcss from 8.5.6 to 8.5.12 in /site (#229)

0.30.1 - 2026-04-22

Documentation

• Updated README with missing CLI commands — Added documentation for recently-added CLI commands that were missing from the README (#220)

0.30.0 - 2026-04-22

Added

• waza quality command — LLM-as-Judge skill quality scoring that evaluates skill output quality using a configurable judge model (#218)
• Scope-reduction advisory check — waza check now includes an advisory that flags skills with overly broad scope, helping authors tighten skill definitions (#219)

0.29.0 - 2026-04-22

Added

• --keep-workspace flag — Preserve the temporary workspace after task execution for debugging agent output (#123, #217)
• --no-skills flag and disabled_skills config — Disable specific skills during evaluation to isolate behavior (#126, #216)
• Non-blocking version update check — CLI now checks for newer waza versions in the background without slowing startup (#104, #214)
• Per-task skill_directories — Specify different skill directories for individual tasks in eval YAML (#156, #215)

Dependencies

• Bump astro and @astrojs/starlight in /site (#212)

0.28.0 - 2026-04-21

Added

• Follow-up prompts in eval YAML — Tasks can now include pre-written follow-up prompts for multi-turn evaluation conversations (#189, #209)
• waza models command — List all available models supported by the configured engine (#208)
• Early termination for trigger tests — Trigger tests can now stop early once the target skill is invoked, reducing evaluation time (#207)

Fixed

• Stricter YAML validation — Audited all YAML parsers; unknown fields in TestCase definitions are now properly rejected (#132, #206)
• Test fixture assertion syntax — Fixed invalid Python expression in a test fixture assertion (#197)
• CI integration test stability — CI integration tests now correctly handle expected eval failures when using the mock executor (#210)

Documentation

• Added Quick Start guide to the documentation site (#205)

0.27.0 - 2026-04-21

Added

• output_contains_any expectation — New expectation field that passes when the agent response contains any one of the specified strings (#203)
• max_response_time_ms behavior rule — Enforce maximum response time constraints on agent execution (#201)
• Task prompt from file — Task prompt field can now reference an external file path instead of inline text (#157, #200)
• tool_calls grader — New grader type that validates the specific tool calls an agent makes during execution (#187, #202)

Fixed

• Webserver test resilience — Webserver tests now skip gracefully when frontend assets are not built (#204)

0.26.0 - 2026-04-21

Changed

• Timestamped output directories — run --output-dir now groups result files by timestamp for cleaner organization (#153)
• Improved debug logging — Debug output is now more structured and useful for troubleshooting (#152)

Fixed

• --discover finds eval.yaml in nested layout — Skill discovery now correctly locates eval.yaml files in evals/{name}/ directories at the project root (#44)
• Diff grader reads post-execution workspace — The diff grader now reads files from the workspace after agent execution completes, not before (#165, #196)
• Grader config validation — Required grader configuration fields are now validated before evaluation starts (#195)
• macOS install and trigger test count — Fixed macOS binary installation and an off-by-one error in trigger test counting (#164, #184, #193)

Documentation

• Added cache command reference, prompt mode documentation, and complete YAML schema reference (#198)
• Updated demo guide and added CI/CD integration guide (#112, #89,

Note truncated.

Fixed

• Tool-call argument grading — tool_calls graders now use canonical tool_events[].args data, preserving MCP and custom tool arguments across results.json round trips (#474, #477).
• Dev command output race — Trigger-discovery warnings are emitted before the Copilot spinner starts, preventing concurrent writes detected by the Go race suite (#506).

Changed

• Documentation site validation — Pull requests now validate the Starlight documentation site before merge (#473, #476).
• GitHub Actions hardening — Third-party actions are pinned to full commit SHAs and Dependabot applies a seven-day GitHub Actions cooldown (#504).
• Dependencies — Updated Playwright, ESLint, TypeScript ESLint, React Query, Starlight, Goldmark, and go-runewidth (#495, #497, #498, #499, #500, #502, #503).

Fixed

• Tool-call argument grading — tool_calls graders now use canonical tool_events[].args data, preserving MCP and custom tool arguments across results.json round trips ( #474 , #477 ).

• Dev command output race — Trigger-discovery warnings are emitted before the Copilot spinner starts, preventing concurrent writes detected by the Go race suite ( #506 ).

Changed

• Documentation site validation — Pull requests now validate the Starlight documentation site before merge ( #473 , #476 ).

• GitHub Actions hardening — Third-party actions are pinned to full commit SHAs and Dependabot applies a seven-day GitHub Actions cooldown ( #504 ).

• Dependencies — Updated Playwright, ESLint, TypeScript ESLint, React Query, Starlight, Goldmark, and go-runewidth ( #495 , #497 , #498 , #499 , #500 , #502 , #503 ).

What's Changed

• chore(deps): bump setup-python action by Shayne Boyer (@spboyer) in #445
• chore(deps): Bump actions/setup-node from 6 to 7 by Dependabot (@dependabot)[bot] in #443
• chore(deps-dev): Bump eslint from 10.6.0 to 10.7.0 in /web by Dependabot (@dependabot)[bot] in #432
• chore(deps-dev): Bump typescript-eslint from 8.62.1 to 8.63.0 in /web by Dependabot (@dependabot)[bot] in #431
• chore(deps): Bump golang.org/x/text from 0.38.0 to 0.40.0 by Dependabot (@dependabot)[bot] in #428
• chore(deps): Bump github.com/azure/azure-dev/cli/azd from 1.27.0 to 1.27.1 by Dependabot (@dependabot)[bot] in #427
• chore(deps): Bump lucide-react from 1.23.0 to 1.24.0 in /web by Dependabot (@dependabot)[bot] in #426
• chore(deps): Bump github.com/github/copilot-sdk/go from 1.0.5 to 1.0.6 by Dependabot (@dependabot)[bot] in #422
• fix: bind Windows update installer URL by Shayne Boyer (@spboyer) in #451
• fix: enforce json_schema extract_json selection by Shayne Boyer (@spboyer) in #469
• fix: expose configured MCP server tools by Shayne Boyer (@spboyer) in #453
• chore(deps): Bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp from 1.43.0 to 1.44.0 by Dependabot (@dependabot)[bot] in #423
• chore(deps): Bump github.com/klauspost/compress from 1.19.0 to 1.19.1 by Dependabot (@dependabot)[bot] in #459
• chore(deps): Bump github.com/github/copilot-sdk/go from 1.0.6 to 1.0.8 by Dependabot (@dependabot)[bot] in #460
• chore(deps): Bump github.com/azure/azure-dev/cli/azd from 1.27.1 to 1.28.1 by Dependabot (@dependabot)[bot] in #464
• chore(deps-dev): Bump brace-expansion from 5.0.6 to 5.0.7 in /web by Dependabot (@dependabot)[bot] in #447
• chore(deps-dev): Bump vite from 8.1.3 to 8.1.5 in /web by Dependabot (@dependabot)[bot] in #465
• chore(deps-dev): Bump @tailwindcss/vite from 4.3.2 to 4.3.3 in /web by Dependabot (@dependabot)[bot] in #466
• chore(deps): Bump svgo from 4.0.1 to 4.0.2 in /site by Dependabot (@dependabot)[bot] in #452
• chore(deps-dev): Bump @playwright/test from 1.61.1 to 1.62.0 in /site by Dependabot (@dependabot)[bot] in #461
• chore(deps): Bump @astrojs/starlight from 0.41.3 to 0.41.4 in /site by Dependabot (@dependabot)[bot] in #462
• chore(deps): Bump actions/setup-go from 6 to 7 by Dependabot (@dependabot)[bot] in #442
• chore(deps): Bump google.golang.org/grpc from 1.81.1 to 1.82.1 by Dependabot (@dependabot)[bot] in #458
• docs: add repository gotchas to copilot-instructions by Shayne Boyer (@spboyer) in #421
• chore(deps): Bump astro from 7.0.6 to 7.1.4 in /site by Dependabot (@dependabot)[bot] in #446
• chore(deps): update site transitive dependencies by Shayne Boyer (@spboyer) in #468
• chore(deps): update root Go dependencies by Shayne Boyer (@spboyer) in #433
• fix: clarify fixture workspace behavior by Shayne Boyer (@spboyer) in #434
• fix: tolerate existing GitHub releases by Shayne Boyer (@spboyer) in #472

Full Changelog: v0.38.3...v0.38.4

Changelog

All notable changes to waza will be documented in this file.

The format is based on Keep a Changelog,
and this project adheres to Semantic Versioning.

Unreleased

Fixed

• Release workflow idempotency — CLI artifact publishing now uploads assets to an existing release when the tag's release was created earlier in the workflow, instead of failing on gh release create.
• Windows update command — waza update now passes the PowerShell installer URL reliably, preventing Invoke-RestMethod from failing with a null or empty Uri (#448).
• JSON schema grader configuration — json_schema graders now accept the documented extract_json option in eval and task YAML, extracting exactly one JSON object or array candidate before schema validation (#457).
• Live MCP server tool availability — config.mcp_servers entries now default omitted tools to ["*"], matching MCP mock behavior so configured stdio and HTTP servers expose their tools to Copilot SDK evaluations (#449).

0.38.3 - 2026-07-17

Fixed

• MCP mock tool availability — mcp_mocks now explicitly exposes declared tools to the bundled Copilot CLI, preventing valid mock servers from being skipped during Copilot SDK evaluations (#440).

0.38.2 - 2026-07-16

Fixed

• JSON schema ambiguity detection — extract_json now rejects output containing a JSON code block plus any additional JSON document.

0.38.1 - 2026-07-16

Fixed

• JSON schema output extraction — The json_schema grader can now extract exactly one JSON document from prose or a Markdown JSON code block with extract_json: true, while retaining strict pure-JSON validation by default.
• APM-installed skill detection — Skill discovery recognizes skills installed by Azure API Management (#402).

Changed

• Dependencies and tooling — Updated the Copilot SDK, OpenTelemetry, azd, documentation tooling, dashboard dependencies, and GitHub Actions.
• Documentation — Added the executor to the getting-started manifest example (#435).

0.38.0 - 2026-06-30

Added

• Focused eval suggestions — waza suggest now supports targeted generation with --count, --focus, --dry-run, --apply, and --force (#357, #380)
• Per-turn checkpoint graders — Task YAML can run inline graders after specific conversation turns with checkpoints[] and on_failure policies (#358, #386)
• Rubric preset library — Prompt graders can reuse built-in rubric presets for common judge dimensions; no separate waza rubric subcommand ships in this release (#360, #381)
• Spec verification — Added waza spec verify to report eval coverage against SKILL.md requirements (#361, #385)
• OpenTelemetry trace export — Added waza run --otel-exporter, --otel-endpoint, --otel-headers, --otel-file, and --otel-include-payloads (#362, #383)
• MCP server mocks — Added eval-level mcp_mocks: for hermetic Copilot SDK tool-call evals (#363, #387)
• Regression gates — Added waza gate with stable exit codes for pass, regression, golden failure, and config errors (#364, #384)
• Adversarial harness — Added waza adversarial and eval-level adversarial: pack configuration for prompt-injection and scope-bypass checks (#365, #392)
• Tool metrics and structured argument matchers — Results now include normalized tool_events[]; tool graders can assert structured argument matchers through expect_tools[].args and tool_calls.expect[].args (#366, #388)
• Snapshot and replay — Added waza run --snapshot and waza replay with a self-contained snapshot artifact format (#367, #391)
• Schema version policy — Documented and enforced MAJOR.MINOR schemaVersion compatibility for public artifacts (#368, #382)
• Dashboard SSE resume — Added Last-Event-ID / lastEventId resume support for dashboard event streams, including legacy /api/events (#178, #397)

Changed

• Phase 1 internal refactor — Internal cleanup with no user-facing CLI, schema, or site behavior changes (#10)

0.37.0 - 2026-06-18

Added

• Interactive skill responder — Eval runs can now drive interactive skills with an LLM responder for more realistic conversational workflows (#304, closes #303)
• Triage automation and regression loop — Added triage automation and regression-loop support for Squad workflow validation (#326)

Fixed

• Task-level context fixtures — Task-level context fixtures now materialize in workspaces before execution (#329)
• waza suggest engine failures — Engine failures are now surfaced by waza suggest instead of being hidden behind success-shaped output (#330)
• Session idle hang — Upgraded the Copilot SDK to v1.0.2 and re-bundled Copilot CLI 1.0.64-0 to fix session idle hangs (#333)

Dependencies

• Bump astro from 6.3.2 to 6.4.7 in /site (#331)
• Bump js-yaml from 4.1.1 to 4.2.0 in /site (#327)

0.36.0 - 2026-06-15

Added

• Squad framework v0.10.0 upgrade — Upgraded Squad from 0.8.25 to 0.10.0 (#322, #323)
• Squad workflow failure detection — Added Squad workflow and failure detection infrastructure (#322, #324)

Fixed

• Prompt grader timeout configuration — Prompt grader timeout can now be configured with WAZA_PROMPT_GRADER_TIMEOUT (#319)
• Session-start hang detection — Added a first-event watchdog to catch session-start hangs (#321)
• Non-Squad coordinator canary handling — Clarified the Squad coordinator canary guard so non-Squad sessions can continue without using Squad (#325)

Dependencies

• Bump esbuild, @tailwindcss/vite, @vitejs/plugin-react, and vite (#317)

0.35.0 - 2026-06-06

Added

• Copilot SDK v1.0.0 upgrade — Upgraded github.com/github/copilot-sdk/go to v1.0.0 and surfaced premium-request credits on the dashboard (#311)
• Model-aware dashboard pricing — Dashboard cost calculation now applies per-model pricing for more accurate run cost reporting (#310)
• Git worktree resources in task inputs — Tasks can now reference git worktree resources as inputs (#121, #302)

Fixed

• BYOK + --model startup arg — The Copilot CLI validates the startup --model flag against the Copilot catalog before BYOK provider config is applied, so provider-only model IDs would fail. The --model startup arg is now skipped when a BYOK provider is configured (#305, #306)
• Model override propagation — --model is now passed via CLIArgs so it correctly overrides user settings and experiment flights (#263)
• Copilot CLI PATH fallback — Prevent silent fallback to a Copilot CLI on PATH when the bundled binary is unavailable (#300)
• Installer latest-release selection — Installer now correctly selects the latest standalone waza release (#299)
• Skill best practices doc link — Fixed the broken skill best practices reference (#295, #298)

Changed

• AgentEngine cancellation — Simplified AgentEngine cancellation handling around caller contexts to make shutdown semantics more predictable (#290)

0.34.0 - 2026-05-23

Added

• BYOK provider wiring — Added bring-your-own-key provider support for configured model providers (#240)
• waza update command — Added an update command for upgrading local Waza installations (#288)
• Skill injection opt-out — Added an option to run evals without injecting the target skill body (#285, #292)
• Forbidden skills grading — skill_invocation graders can now assert that specific skills must not be invoked (#286, #291)
• Per-trial usage reporting — Results JSON now includes per-trial usage details for deeper run analysis (#277)
• Agent-friendly GitHub templates — Added issue and pull request templates tuned for agent-authored work (#293)

Fixed

• Tool approval handling — Tool permission handling now uses the SDK approval kind (#240)
• Signal cancellation — waza run now respects cancellation signals more reliably (#279)
• Sandbox prompt handling — Empty sandbox prompts are guarded before execution (#273, #278)
• Custom agent example schema — Fixed the custom-agent eval example to match the supported schema (#282)
• Binary release links — Fixed binary release documentation links (#276, #284)
• Agent path guidance — Corrected AGENTS root path guidance (#267, #269)

Changed

• Run concurrency — waza run now reuses a shared Copilot client and auto-sizes parallel workers when --workers is unset (#135, #221)
• Documentation — Updated integration testing, custom-agent eval, and OpenAI Evals model-graded YAML documentation (#281, #283, #14, #280)
• Release workflow — GitHub Pages deployment now runs after the release workflow (#265)

0.33.0 - 2026-05-21

Note: This release includes the changes previously prepared under 0.32.0, which was not published.

Added

• Configurable eval file naming — .waza.yaml can now configure files.evalFile, files.taskGlob, and files.taskFileSuffix, with the new naming carried through scaffolding, workspace discovery, discovery mode, schemas, and docs while preserving the existing eval.yaml and tasks/*.yaml defaults (#254, closes #232)
• Instruction files in eval runs — Eval-level config.instruction_files and task-level instruction_files now copy files from the active context into task workspaces and append path-labeled contents to the Copilot system message (#248, closes #239)

Fixed

• Prompt graders use the execution engine — Prompt graders now route judge turns through CopilotEngine instead of constructing a Copilot client directly, keeping grader execution aligned with engine configuration and preserving follow-up recovery behavior (#258, closes #54)
• Prompt grader follow-up recovery — Prompt grading now preserves collected grades when a follow-up turn fails after successful grader collection (#251)
• Bundled Copilot CLI updated — Embedded copilot-cli bundles are updated from 1.0.2 to 1.0.49 across supported platforms, with reproducible pinned bundle generation via COPILOT_CLI_VERSION (#260, closes #244)
• Spec-aligned skill scaffolding — waza new skill no longer asks for a nonstandard skill type or emits type: frontmatter, and the wizard now rejects early exits that omit required name or description fields (#261, closes #243)
• waza check eval discovery — Nested skills and separated evals are discovered consistently in multi-skill workspaces (#247, closes #238)
• Skill body routing markers — Compliance scoring now detects trigger, anti-trigger, and routing markers in SKILL.md body sections as well as frontmatter descriptions (#236, closes #223)

Changed

• Copilot SDK v0.3.0 migration — Updated github.com/github/copilot-sdk/go to v0.3.0, migrated session event handling to typed payloads, and refreshed transcript, logging, web API, usage collection, suggestion trace, and test coverage for the new API (#255, closes #253)
• Dashboard validation coverage — Added coverage for dashboard lint and end-to-end validation (#249)
• Install documentation — Replaced unsupported go install guidance and clarified Windows/WSL install behavior (#246, closes #242; #245, closes #241)
• Dependencies — Bump devalue in /site, postcss in /web, and astro in /site (#237, #235, #234)

0.31.0 - 2026-04-28

Added

• Custom agent (.agent.md) eval support — Discover .agent.md files alongside SKILL.md, parse agent-specific frontmatter (tools, model, handoffs, mcp-servers, agents), auto-inject tool_constraint grader from agent tools: field, complete worked example under examples/custom-agent/, and new "Evaluating Custom Agents" docs guide (#226, closes #225)

Fixed

• Mock engine echoes file content — _output_contains expectations against file contents now work in CI without a real model. Mock response includes task metadata, file paths, and a 1KB content preview per resource (#228, closes #227)
• waza serve no longer crashes when stdin isn't a terminal — MCP stdio server only starts when term.IsTerminal() is true; piped input or background mode no longer kills the HTTP dashboard (#224)

Changed

• Vocabulary renames — Internal types renamed: BenchmarkSpec → EvalSpec, TestRunner → EvalRunner. Not a breaking change for external consumers (types live in internal/) (#222)

Documentation

• Cross-reference audit for recent renames + custom agent feature: added .agent.md coverage to quickstart, getting-started, GUIDE, TUTORIAL, examples README; updated mock engine descriptions in INTEGRATION-TESTING and eval-yaml guide (#230)

Dependencies

• Bump postcss from 8.5.6 to 8.5.12 in /site (#229)

0.30.1 - 2026-04-22

Documentation

• Updated README with missing CLI commands — Added documentation for recently-added CLI commands that were missing from the README (#220)

0.30.0 - 2026-04-22

Added

• waza quality command — LLM-as-Judge skill quality scoring that evaluates skill output quality using a configurable judge model (#218)
• Scope-reduction advisory check — waza check now includes an advisory that flags skills with overly broad scope, helping authors tighten skill definitions (#219)

0.29.0 - 2026-04-22

Added

• --keep-workspace flag — Preserve the temporary workspace after task execution for debugging agent output (#123, #217)
• --no-skills flag and disabled_skills config — Disable specific skills during evaluation to isolate behavior (#126, #216)
• Non-blocking version update check — CLI now checks for newer waza versions in the background without slowing startup (#104, #214)
• Per-task skill_directories — Specify different skill directories for individual tasks in eval YAML (#156, #215)

Dependencies

• Bump astro and @astrojs/starlight in /site (#212)

0.28.0 - 2026-04-21

Added

• Follow-up prompts in eval YAML — Tasks can now include pre-written follow-up prompts for multi-turn evaluation conversations (#189, #209)
• waza models command — List all available models supported by the configured engine (#208)
• Early termination for trigger tests — Trigger tests can now stop early once the target skill is invoked, reducing evaluation time (#207)

Fixed

• Stricter YAML validation — Audited all YAML parsers; unknown fields in TestCase definitions are now properly rejected (#132, #206)
• Test fixture assertion syntax — Fixed invalid Python expression in a test fixture assertion (#197)
• CI integration test stability — CI integration tests now correctly handle expected eval failures when using the mock executor (#210)

Documentation

• Added Quick Start guide to the documentation site (#205)

0.27.0 - 2026-04-21

Added

• output_contains_any expectation — New expectation field that passes when the agent response contains any one of the specified strings (#203)
• max_response_time_ms behavior rule — Enforce maximum response time constraints on agent execution (#201)
• Task prompt from file — Task prompt field can now reference an external file path instead of inline text (#157, #200)
• tool_calls grader — New grader type that validates the specific tool calls an agent makes during execution (#187, #202)

Fixed

• Webserver test resilience — Webserver tests now skip gracefully when frontend assets are not built (#204)

0.26.0 - 2026-04-21

Changed

• Timestamped output directories — run --output-dir now groups result files by timestamp for cleaner organization (#153)
• Improved debug logging — Debug output is now more structured and useful for troubleshooting (#152)

Fixed

• --discover finds eval.yaml in nested layout — Skill discovery now correctly locates eval.yaml files in evals/{name}/ directories at the project root (#44)
• Diff grader reads post-execution workspace — The diff grader now reads files from the workspace after agent execution completes, not before (#165, #196)
• Grader config validation — Required grader configuration fields are now validated before evaluation starts (#195)
• macOS install and trigger test count — Fixed macOS binary installation and an off-by-one error in trigger test counting (#164, #184, #193)

Documentation

• Added cache command reference, prompt mode documentation, and complete YAML schema reference (#198)
• Updated demo guide and added CI/CD integration guide (#112, #89, #194)

Dependencies

• Bump defu from 6.1.4 to 6.1.6 in /site (#181)
• Bump vite from 6.4.1 to 6.4.2 in /site and /web (#182, #192)
• Bump go.opentelemetry.io/otel/sdk from 1.42.0 to 1.43.0 (#185)
• Bump astro from 5.17.3 to 5.18.1 in /site (#163)
• Bump picomatch from 4.0.3 to 4.0.4 in /site and /web (#159, #160)
• Bump smol-toml from 1.6.0 to 1.6.1 in /site (#158)

0.25.0 - 2026-04-21

Added

• Eval coverage grid generator — New coverage output that visualizes which skills have eval coverage across grader types (#92)

Fixed

• SKILL.md injection and trigger fixture loading — waza run now correctly injects SKILL.md content into the evaluation context, loads trigger test fixtures, and passes MCP server configuration to the engine (#191)

Dependencies

• Bump h3 from 1.15.5 to 1.15.8 in /site (#144)

0.24.0 - 2026-03-25

Changed

• Strict YAML validation — All YAML parsers now use KnownFields(true) to reject unknown fields, catching typos and misconfigurations early (#132,

Note truncated.

Fixed

• Release workflow idempotency — CLI artifact publishing now uploads assets to an existing release when the tag's release was created earlier in the workflow, instead of failing on gh release create.
• Windows update command — waza update now passes the PowerShell installer URL reliably, preventing Invoke-RestMethod from failing with a null or empty Uri (#448).
• JSON schema grader configuration — json_schema graders now accept the documented extract_json option in eval and task YAML, extracting exactly one JSON object or array candidate before schema validation (#457).
• Live MCP server tool availability — config.mcp_servers entries now default omitted tools to ["*"], matching MCP mock behavior so configured stdio and HTTP servers expose their tools to Copilot SDK evaluations (#449).

Fixed

• Release workflow idempotency — CLI artifact publishing now uploads assets to an existing release when the tag's release was created earlier in the workflow, instead of failing on gh release create .

• Windows update command — waza update now passes the PowerShell installer URL reliably, preventing Invoke-RestMethod from failing with a null or empty Uri ( #448 ).

• JSON schema grader configuration — json_schema graders now accept the documented extract_json option in eval and task YAML, extracting exactly one JSON object or array candidate before schema validation ( #457 ).

• Live MCP server tool availability — config.mcp_servers entries now default omitted tools to ["*"] , matching MCP mock behavior so configured stdio and HTTP servers expose their tools to Copilot SDK evaluations ( #449 ).

What's Changed

• fix: expose MCP mock tools to Copilot CLI by Shayne Boyer (@spboyer) in #441

Full Changelog: v0.38.2...v0.38.3

Changelog

All notable changes to waza will be documented in this file.

The format is based on Keep a Changelog,
and this project adheres to Semantic Versioning.

Unreleased

0.38.3 - 2026-07-17

Fixed

• MCP mock tool availability — mcp_mocks now explicitly exposes declared tools to the bundled Copilot CLI, preventing valid mock servers from being skipped during Copilot SDK evaluations (#440).

0.38.2 - 2026-07-16

Fixed

• JSON schema ambiguity detection — extract_json now rejects output containing a JSON code block plus any additional JSON document.

0.38.1 - 2026-07-16

Fixed

• JSON schema output extraction — The json_schema grader can now extract exactly one JSON document from prose or a Markdown JSON code block with extract_json: true, while retaining strict pure-JSON validation by default.
• APM-installed skill detection — Skill discovery recognizes skills installed by Azure API Management (#402).

Changed

• Dependencies and tooling — Updated the Copilot SDK, OpenTelemetry, azd, documentation tooling, dashboard dependencies, and GitHub Actions.
• Documentation — Added the executor to the getting-started manifest example (#435).

0.38.0 - 2026-06-30

Added

• Focused eval suggestions — waza suggest now supports targeted generation with --count, --focus, --dry-run, --apply, and --force (#357, #380)
• Per-turn checkpoint graders — Task YAML can run inline graders after specific conversation turns with checkpoints[] and on_failure policies (#358, #386)
• Rubric preset library — Prompt graders can reuse built-in rubric presets for common judge dimensions; no separate waza rubric subcommand ships in this release (#360, #381)
• Spec verification — Added waza spec verify to report eval coverage against SKILL.md requirements (#361, #385)
• OpenTelemetry trace export — Added waza run --otel-exporter, --otel-endpoint, --otel-headers, --otel-file, and --otel-include-payloads (#362, #383)
• MCP server mocks — Added eval-level mcp_mocks: for hermetic Copilot SDK tool-call evals (#363, #387)
• Regression gates — Added waza gate with stable exit codes for pass, regression, golden failure, and config errors (#364, #384)
• Adversarial harness — Added waza adversarial and eval-level adversarial: pack configuration for prompt-injection and scope-bypass checks (#365, #392)
• Tool metrics and structured argument matchers — Results now include normalized tool_events[]; tool graders can assert structured argument matchers through expect_tools[].args and tool_calls.expect[].args (#366, #388)
• Snapshot and replay — Added waza run --snapshot and waza replay with a self-contained snapshot artifact format (#367, #391)
• Schema version policy — Documented and enforced MAJOR.MINOR schemaVersion compatibility for public artifacts (#368, #382)
• Dashboard SSE resume — Added Last-Event-ID / lastEventId resume support for dashboard event streams, including legacy /api/events (#178, #397)

Changed

• Phase 1 internal refactor — Internal cleanup with no user-facing CLI, schema, or site behavior changes (#10)

0.37.0 - 2026-06-18

Added

• Interactive skill responder — Eval runs can now drive interactive skills with an LLM responder for more realistic conversational workflows (#304, closes #303)
• Triage automation and regression loop — Added triage automation and regression-loop support for Squad workflow validation (#326)

Fixed

• Task-level context fixtures — Task-level context fixtures now materialize in workspaces before execution (#329)
• waza suggest engine failures — Engine failures are now surfaced by waza suggest instead of being hidden behind success-shaped output (#330)
• Session idle hang — Upgraded the Copilot SDK to v1.0.2 and re-bundled Copilot CLI 1.0.64-0 to fix session idle hangs (#333)

Dependencies

• Bump astro from 6.3.2 to 6.4.7 in /site (#331)
• Bump js-yaml from 4.1.1 to 4.2.0 in /site (#327)

0.36.0 - 2026-06-15

Added

• Squad framework v0.10.0 upgrade — Upgraded Squad from 0.8.25 to 0.10.0 (#322, #323)
• Squad workflow failure detection — Added Squad workflow and failure detection infrastructure (#322, #324)

Fixed

• Prompt grader timeout configuration — Prompt grader timeout can now be configured with WAZA_PROMPT_GRADER_TIMEOUT (#319)
• Session-start hang detection — Added a first-event watchdog to catch session-start hangs (#321)
• Non-Squad coordinator canary handling — Clarified the Squad coordinator canary guard so non-Squad sessions can continue without using Squad (#325)

Dependencies

• Bump esbuild, @tailwindcss/vite, @vitejs/plugin-react, and vite (#317)

0.35.0 - 2026-06-06

Added

• Copilot SDK v1.0.0 upgrade — Upgraded github.com/github/copilot-sdk/go to v1.0.0 and surfaced premium-request credits on the dashboard (#311)
• Model-aware dashboard pricing — Dashboard cost calculation now applies per-model pricing for more accurate run cost reporting (#310)
• Git worktree resources in task inputs — Tasks can now reference git worktree resources as inputs (#121, #302)

Fixed

• BYOK + --model startup arg — The Copilot CLI validates the startup --model flag against the Copilot catalog before BYOK provider config is applied, so provider-only model IDs would fail. The --model startup arg is now skipped when a BYOK provider is configured (#305, #306)
• Model override propagation — --model is now passed via CLIArgs so it correctly overrides user settings and experiment flights (#263)
• Copilot CLI PATH fallback — Prevent silent fallback to a Copilot CLI on PATH when the bundled binary is unavailable (#300)
• Installer latest-release selection — Installer now correctly selects the latest standalone waza release (#299)
• Skill best practices doc link — Fixed the broken skill best practices reference (#295, #298)

Changed

• AgentEngine cancellation — Simplified AgentEngine cancellation handling around caller contexts to make shutdown semantics more predictable (#290)

0.34.0 - 2026-05-23

Added

• BYOK provider wiring — Added bring-your-own-key provider support for configured model providers (#240)
• waza update command — Added an update command for upgrading local Waza installations (#288)
• Skill injection opt-out — Added an option to run evals without injecting the target skill body (#285, #292)
• Forbidden skills grading — skill_invocation graders can now assert that specific skills must not be invoked (#286, #291)
• Per-trial usage reporting — Results JSON now includes per-trial usage details for deeper run analysis (#277)
• Agent-friendly GitHub templates — Added issue and pull request templates tuned for agent-authored work (#293)

Fixed

• Tool approval handling — Tool permission handling now uses the SDK approval kind (#240)
• Signal cancellation — waza run now respects cancellation signals more reliably (#279)
• Sandbox prompt handling — Empty sandbox prompts are guarded before execution (#273, #278)
• Custom agent example schema — Fixed the custom-agent eval example to match the supported schema (#282)
• Binary release links — Fixed binary release documentation links (#276, #284)
• Agent path guidance — Corrected AGENTS root path guidance (#267, #269)

Changed

• Run concurrency — waza run now reuses a shared Copilot client and auto-sizes parallel workers when --workers is unset (#135, #221)
• Documentation — Updated integration testing, custom-agent eval, and OpenAI Evals model-graded YAML documentation (#281, #283, #14, #280)
• Release workflow — GitHub Pages deployment now runs after the release workflow (#265)

0.33.0 - 2026-05-21

Note: This release includes the changes previously prepared under 0.32.0, which was not published.

Added

• Configurable eval file naming — .waza.yaml can now configure files.evalFile, files.taskGlob, and files.taskFileSuffix, with the new naming carried through scaffolding, workspace discovery, discovery mode, schemas, and docs while preserving the existing eval.yaml and tasks/*.yaml defaults (#254, closes #232)
• Instruction files in eval runs — Eval-level config.instruction_files and task-level instruction_files now copy files from the active context into task workspaces and append path-labeled contents to the Copilot system message (#248, closes #239)

Fixed

• Prompt graders use the execution engine — Prompt graders now route judge turns through CopilotEngine instead of constructing a Copilot client directly, keeping grader execution aligned with engine configuration and preserving follow-up recovery behavior (#258, closes #54)
• Prompt grader follow-up recovery — Prompt grading now preserves collected grades when a follow-up turn fails after successful grader collection (#251)
• Bundled Copilot CLI updated — Embedded copilot-cli bundles are updated from 1.0.2 to 1.0.49 across supported platforms, with reproducible pinned bundle generation via COPILOT_CLI_VERSION (#260, closes #244)
• Spec-aligned skill scaffolding — waza new skill no longer asks for a nonstandard skill type or emits type: frontmatter, and the wizard now rejects early exits that omit required name or description fields (#261, closes #243)
• waza check eval discovery — Nested skills and separated evals are discovered consistently in multi-skill workspaces (#247, closes #238)
• Skill body routing markers — Compliance scoring now detects trigger, anti-trigger, and routing markers in SKILL.md body sections as well as frontmatter descriptions (#236, closes #223)

Changed

• Copilot SDK v0.3.0 migration — Updated github.com/github/copilot-sdk/go to v0.3.0, migrated session event handling to typed payloads, and refreshed transcript, logging, web API, usage collection, suggestion trace, and test coverage for the new API (#255, closes #253)
• Dashboard validation coverage — Added coverage for dashboard lint and end-to-end validation (#249)
• Install documentation — Replaced unsupported go install guidance and clarified Windows/WSL install behavior (#246, closes #242; #245, closes #241)
• Dependencies — Bump devalue in /site, postcss in /web, and astro in /site (#237, #235, #234)

0.31.0 - 2026-04-28

Added

• Custom agent (.agent.md) eval support — Discover .agent.md files alongside SKILL.md, parse agent-specific frontmatter (tools, model, handoffs, mcp-servers, agents), auto-inject tool_constraint grader from agent tools: field, complete worked example under examples/custom-agent/, and new "Evaluating Custom Agents" docs guide (#226, closes #225)

Fixed

• Mock engine echoes file content — _output_contains expectations against file contents now work in CI without a real model. Mock response includes task metadata, file paths, and a 1KB content preview per resource (#228, closes #227)
• waza serve no longer crashes when stdin isn't a terminal — MCP stdio server only starts when term.IsTerminal() is true; piped input or background mode no longer kills the HTTP dashboard (#224)

Changed

• Vocabulary renames — Internal types renamed: BenchmarkSpec → EvalSpec, TestRunner → EvalRunner. Not a breaking change for external consumers (types live in internal/) (#222)

Documentation

• Cross-reference audit for recent renames + custom agent feature: added .agent.md coverage to quickstart, getting-started, GUIDE, TUTORIAL, examples README; updated mock engine descriptions in INTEGRATION-TESTING and eval-yaml guide (#230)

Dependencies

• Bump postcss from 8.5.6 to 8.5.12 in /site (#229)

0.30.1 - 2026-04-22

Documentation

• Updated README with missing CLI commands — Added documentation for recently-added CLI commands that were missing from the README (#220)

0.30.0 - 2026-04-22

Added

• waza quality command — LLM-as-Judge skill quality scoring that evaluates skill output quality using a configurable judge model (#218)
• Scope-reduction advisory check — waza check now includes an advisory that flags skills with overly broad scope, helping authors tighten skill definitions (#219)

0.29.0 - 2026-04-22

Added

• --keep-workspace flag — Preserve the temporary workspace after task execution for debugging agent output (#123, #217)
• --no-skills flag and disabled_skills config — Disable specific skills during evaluation to isolate behavior (#126, #216)
• Non-blocking version update check — CLI now checks for newer waza versions in the background without slowing startup (#104, #214)
• Per-task skill_directories — Specify different skill directories for individual tasks in eval YAML (#156, #215)

Dependencies

• Bump astro and @astrojs/starlight in /site (#212)

0.28.0 - 2026-04-21

Added

• Follow-up prompts in eval YAML — Tasks can now include pre-written follow-up prompts for multi-turn evaluation conversations (#189, #209)
• waza models command — List all available models supported by the configured engine (#208)
• Early termination for trigger tests — Trigger tests can now stop early once the target skill is invoked, reducing evaluation time (#207)

Fixed

• Stricter YAML validation — Audited all YAML parsers; unknown fields in TestCase definitions are now properly rejected (#132, #206)
• Test fixture assertion syntax — Fixed invalid Python expression in a test fixture assertion (#197)
• CI integration test stability — CI integration tests now correctly handle expected eval failures when using the mock executor (#210)

Documentation

• Added Quick Start guide to the documentation site (#205)

0.27.0 - 2026-04-21

Added

• output_contains_any expectation — New expectation field that passes when the agent response contains any one of the specified strings (#203)
• max_response_time_ms behavior rule — Enforce maximum response time constraints on agent execution (#201)
• Task prompt from file — Task prompt field can now reference an external file path instead of inline text (#157, #200)
• tool_calls grader — New grader type that validates the specific tool calls an agent makes during execution (#187, #202)

Fixed

• Webserver test resilience — Webserver tests now skip gracefully when frontend assets are not built (#204)

0.26.0 - 2026-04-21

Changed

• Timestamped output directories — run --output-dir now groups result files by timestamp for cleaner organization (#153)
• Improved debug logging — Debug output is now more structured and useful for troubleshooting (#152)

Fixed

• --discover finds eval.yaml in nested layout — Skill discovery now correctly locates eval.yaml files in evals/{name}/ directories at the project root (#44)
• Diff grader reads post-execution workspace — The diff grader now reads files from the workspace after agent execution completes, not before (#165, #196)
• Grader config validation — Required grader configuration fields are now validated before evaluation starts (#195)
• macOS install and trigger test count — Fixed macOS binary installation and an off-by-one error in trigger test counting (#164, #184, #193)

Documentation

• Added cache command reference, prompt mode documentation, and complete YAML schema reference (#198)
• Updated demo guide and added CI/CD integration guide (#112, #89, #194)

Dependencies

• Bump defu from 6.1.4 to 6.1.6 in /site (#181)
• Bump vite from 6.4.1 to 6.4.2 in /site and /web (#182, #192)
• Bump go.opentelemetry.io/otel/sdk from 1.42.0 to 1.43.0 (#185)
• Bump astro from 5.17.3 to 5.18.1 in /site (#163)
• Bump picomatch from 4.0.3 to 4.0.4 in /site and /web (#159, #160)
• Bump smol-toml from 1.6.0 to 1.6.1 in /site (#158)

0.25.0 - 2026-04-21

Added

• Eval coverage grid generator — New coverage output that visualizes which skills have eval coverage across grader types (#92)

Fixed

• SKILL.md injection and trigger fixture loading — waza run now correctly injects SKILL.md content into the evaluation context, loads trigger test fixtures, and passes MCP server configuration to the engine (#191)

Dependencies

• Bump h3 from 1.15.5 to 1.15.8 in /site (#144)

0.24.0 - 2026-03-25

Changed

• Strict YAML validation — All YAML parsers now use KnownFields(true) to reject unknown fields, catching typos and misconfigurations early (#132, #133)
• max_workers renamed to workers — Config YAML key renamed for consistency across all config types (breaking change)
• Unified token counting — waza check and waza tokens count now share the same counting logic for consistent results (#146)

Fixed

• Typo in prompt grader — Fixed "prmopt" → "prompt" in error message

Dependencies

• Bump h3 from 1.15.8 to 1.15.9 in /site (#155)
• Bump github.com/buger/jsonparser from 1.1.1 to 1.1.2 (#149)

Note truncated.

Fixed

• MCP mock tool availability — mcp_mocks now explicitly exposes declared tools to the bundled Copilot CLI, preventing valid mock servers from being skipped during Copilot SDK evaluations (#440).

Fixed

• MCP mock tool availability — mcp_mocks now explicitly exposes declared tools to the bundled Copilot CLI, preventing valid mock servers from being skipped during Copilot SDK evaluations ( #440 ).

Fixed

• JSON schema ambiguity detection — extract_json now rejects output containing a JSON code block plus any additional JSON document.

Fixed

• JSON schema ambiguity detection — extract_json now rejects output containing a JSON code block plus any additional JSON document.

Fixed

• JSON schema output extraction — The json_schema grader can now extract exactly one JSON document from prose or a Markdown JSON code block with extract_json: true, while retaining strict pure-JSON validation by default.
• APM-installed skill detection — Skill discovery recognizes skills installed by Azure API Management (#402).

Changed

• Dependencies and tooling — Updated the Copilot SDK, OpenTelemetry, azd, documentation tooling, dashboard dependencies, and GitHub Actions.
• Documentation — Added the executor to the getting-started manifest example (#435).

Fixed

• JSON schema output extraction — The json_schema grader can now extract exactly one JSON document from prose or a Markdown JSON code block with extract_json: true , while retaining strict pure-JSON validation by default.

• APM-installed skill detection — Skill discovery recognizes skills installed by Azure API Management ( #402 ).

Changed

• Dependencies and tooling — Updated the Copilot SDK, OpenTelemetry, azd, documentation tooling, dashboard dependencies, and GitHub Actions.

• Documentation — Added the executor to the getting-started manifest example ( #435 ).

Added

• Focused eval suggestions — waza suggest now supports targeted generation with --count, --focus, --dry-run, --apply, and --force (#357, #380)
• Per-turn checkpoint graders — Task YAML can run inline graders after specific conversation turns with checkpoints[] and on_failure policies (#358, #386)
• Rubric preset library — Prompt graders can reuse built-in rubric presets for common judge dimensions; no separate waza rubric subcommand ships in this release (#360, #381)
• Spec verification — Added waza spec verify to report eval coverage against SKILL.md requirements (#361, #385)
• OpenTelemetry trace export — Added waza run --otel-exporter, --otel-endpoint, --otel-headers, --otel-file, and --otel-include-payloads (#362, #383)
• MCP server mocks — Added eval-level mcp_mocks: for hermetic Copilot SDK tool-call evals (#363, #387)
• Regression gates — Added waza gate with stable exit codes for pass, regression, golden failure, and config errors (#364, #384)
• Adversarial harness — Added waza adversarial and eval-level adversarial: pack configuration for prompt-injection and scope-bypass checks (#365, #392)
• Tool metrics and structured argument matchers — Results now include normalized tool_events[]; tool graders can assert structured argument matchers through expect_tools[].args and tool_calls.expect[].args (#366, #388)
• Snapshot and replay — Added waza run --snapshot and waza replay with a self-contained snapshot artifact format (#367, #391)
• Schema version policy — Documented and enforced MAJOR.MINOR schemaVersion compatibility for public artifacts (#368, #382)
• Dashboard SSE resume — Added Last-Event-ID / lastEventId resume support for dashboard event streams, including legacy /api/events (#178, #397)

Changed

• Phase 1 internal refactor — Internal cleanup with no user-facing CLI, schema, or site behavior changes (#10)

Added

• Focused eval suggestions — waza suggest now supports targeted generation with --count , --focus , --dry-run , --apply , and --force ( #357 , #380 )

• Per-turn checkpoint graders — Task YAML can run inline graders after specific conversation turns with checkpoints[] and on_failure policies ( #358 , #386 )

• Rubric preset library — Prompt graders can reuse built-in rubric presets for common judge dimensions; no separate waza rubric subcommand ships in this release ( #360 , #381 )

• Spec verification — Added waza spec verify to report eval coverage against SKILL.md requirements ( #361 , #385 )

• OpenTelemetry trace export — Added waza run --otel-exporter , --otel-endpoint , --otel-headers , --otel-file , and --otel-include-payloads ( #362 , #383 )

• MCP server mocks — Added eval-level mcp_mocks: for hermetic Copilot SDK tool-call evals ( #363 , #387 )

• Regression gates — Added waza gate with stable exit codes for pass, regression, golden failure, and config errors ( #364 , #384 )

• Adversarial harness — Added waza adversarial and eval-level adversarial: pack configuration for prompt-injection and scope-bypass checks ( #365 , #392 )

• Tool metrics and structured argument matchers — Results now include normalized tool_events[] ; tool graders can assert structured argument matchers through expect_tools[].args and tool_calls.expect[].args ( #366 , #388 )

• Snapshot and replay — Added waza run --snapshot and waza replay with a self-contained snapshot artifact format ( #367 , #391 )

• Schema version policy — Documented and enforced MAJOR.MINOR schemaVersion compatibility for public artifacts ( #368 , #382 )

• Dashboard SSE resume — Added Last-Event-ID / lastEventId resume support for dashboard event streams, including legacy /api/events ( #178 , #397 )

Changed

• Phase 1 internal refactor — Internal cleanup with no user-facing CLI, schema, or site behavior changes ( #10 )

Added

• Interactive skill responder — Eval runs can now drive interactive skills with an LLM responder for more realistic conversational workflows (#304, closes #303)
• Triage automation and regression loop — Added triage automation and regression-loop support for Squad workflow validation (#326)

Fixed

• Task-level context fixtures — Task-level context fixtures now materialize in workspaces before execution (#329)
• waza suggest engine failures — Engine failures are now surfaced by waza suggest instead of being hidden behind success-shaped output (#330)
• Session idle hang — Upgraded the Copilot SDK to v1.0.2 and re-bundled Copilot CLI 1.0.64-0 to fix session idle hangs (#333)

Dependencies

• Bump astro from 6.3.2 to 6.4.7 in /site (#331)
• Bump js-yaml from 4.1.1 to 4.2.0 in /site (#327)

Added

• Interactive skill responder — Eval runs can now drive interactive skills with an LLM responder for more realistic conversational workflows ( #304 , closes #303 )

• Triage automation and regression loop — Added triage automation and regression-loop support for Squad workflow validation ( #326 )

Fixed

• Task-level context fixtures — Task-level context fixtures now materialize in workspaces before execution ( #329 )

• waza suggest engine failures — Engine failures are now surfaced by waza suggest instead of being hidden behind success-shaped output ( #330 )

• Session idle hang — Upgraded the Copilot SDK to v1.0.2 and re-bundled Copilot CLI 1.0.64-0 to fix session idle hangs ( #333 )

Dependencies

• Bump astro from 6.3.2 to 6.4.7 in /site ( #331 )

• Bump js-yaml from 4.1.1 to 4.2.0 in /site ( #327 )

Note: This release includes the changes previously prepared under 0.32.0, which was not published.

Added

• Configurable eval file naming — .waza.yaml can now configure files.evalFile, files.taskGlob, and files.taskFileSuffix, with the new naming carried through scaffolding, workspace discovery, discovery mode, schemas, and docs while preserving the existing eval.yaml and tasks/*.yaml defaults (#254, closes #232)
• Instruction files in eval runs — Eval-level config.instruction_files and task-level instruction_files now copy files from the active context into task workspaces and append path-labeled contents to the Copilot system message (#248, closes #239)

Fixed

• Prompt graders use the execution engine — Prompt graders now route judge turns through CopilotEngine instead of constructing a Copilot client directly, keeping grader execution aligned with engine configuration and preserving follow-up recovery behavior (#258, closes #54)
• Prompt grader follow-up recovery — Prompt grading now preserves collected grades when a follow-up turn fails after successful grader collection (#251)
• Bundled Copilot CLI updated — Embedded copilot-cli bundles are updated from 1.0.2 to 1.0.49 across supported platforms, with reproducible pinned bundle generation via COPILOT_CLI_VERSION (#260, closes #244)
• Spec-aligned skill scaffolding — waza new skill no longer asks for a nonstandard skill type or emits type: frontmatter, and the wizard now rejects early exits that omit required name or description fields (#261, closes #243)
• waza check eval discovery — Nested skills and separated evals are discovered consistently in multi-skill workspaces (#247, closes #238)
• Skill body routing markers — Compliance scoring now detects trigger, anti-trigger, and routing markers in SKILL.md body sections as well as frontmatter descriptions (#236, closes #223)

Changed

• Copilot SDK v0.3.0 migration — Updated github.com/github/copilot-sdk/go to v0.3.0, migrated session event handling to typed payloads, and refreshed transcript, logging, web API, usage collection, suggestion trace, and test coverage for the new API (#255, closes #253)
• Dashboard validation coverage — Added coverage for dashboard lint and end-to-end validation (#249)
• Install documentation — Replaced unsupported go install guidance and clarified Windows/WSL install behavior (#246, closes #242; #245, closes #241)
• Dependencies — Bump devalue in /site, postcss in /web, and astro in /site (#237, #235, #234)

Note: This release includes the changes previously prepared under 0.32.0, which was not published.

Added

• Configurable eval file naming — .waza.yaml can now configure files.evalFile , files.taskGlob , and files.taskFileSuffix , with the new naming ca...

Read more

Assets 8 Loading

Uh oh!

There was an error while loading. Please reload this page .

All reactions

Added

• Custom agent (.agent.md) eval support — Discover .agent.md files alongside SKILL.md, parse agent-specific frontmatter (tools, model, handoffs, mcp-servers, agents), auto-inject tool_constraint grader from agent tools: field, complete worked example under examples/custom-agent/, and new "Evaluating Custom Agents" docs guide (#226, closes #225)

Fixed

• Mock engine echoes file content — _output_contains expectations against file contents now work in CI without a real model. Mock response includes task metadata, file paths, and a 1KB content preview per resource (#228, closes #227)
• waza serve no longer crashes when stdin isn't a terminal — MCP stdio server only starts when term.IsTerminal() is true; piped input or background mode no longer kills the HTTP dashboard (#224)

Changed

• Vocabulary renames — Internal types renamed: BenchmarkSpec → EvalSpec, TestRunner → EvalRunner. Not a breaking change for external consumers (types live in internal/) (#222)

Documentation

• Cross-reference audit for recent renames + custom agent feature: added .agent.md coverage to quickstart, getting-started, GUIDE, TUTORIAL, examples README; updated mock engine descriptions in INTEGRATION-TESTING and eval-yaml guide (#230)

Dependencies

• Bump postcss from 8.5.6 to 8.5.12 in /site (#229)

Documentation

• Updated README with missing CLI commands — Added documentation for recently-added CLI commands that were missing from the README (#220)

Added

• waza quality command — LLM-as-Judge skill quality scoring that evaluates skill output quality using a configurable judge model (#218)
• Scope-reduction advisory check — waza check now includes an advisory that flags skills with overly broad scope, helping authors tighten skill definitions (#219)

Added

• --keep-workspace flag — Preserve the temporary workspace after task execution for debugging agent output (#123, #217)
• --no-skills flag and disabled_skills config — Disable specific skills during evaluation to isolate behavior (#126, #216)
• Non-blocking version update check — CLI now checks for newer waza versions in the background without slowing startup (#104, #214)
• Per-task skill_directories — Specify different skill directories for individual tasks in eval YAML (#156, #215)

Dependencies

• Bump astro and @astrojs/starlight in /site (#212)

Added

• Follow-up prompts in eval YAML — Tasks can now include pre-written follow-up prompts for multi-turn evaluation conversations (#189, #209)
• waza models command — List all available models supported by the configured engine (#208)
• Early termination for trigger tests — Trigger tests can now stop early once the target skill is invoked, reducing evaluation time (#207)

Fixed

• Stricter YAML validation — Audited all YAML parsers; unknown fields in TestCase definitions are now properly rejected (#132, #206)
• Test fixture assertion syntax — Fixed invalid Python expression in a test fixture assertion (#197)
• CI integration test stability — CI integration tests now correctly handle expected eval failures when using the mock executor (#210)

Documentation

• Added Quick Start guide to the documentation site (#205)

Added

• output_contains_any expectation — New expectation field that passes when the agent response contains any one of the specified strings (#203)
• max_response_time_ms behavior rule — Enforce maximum response time constraints on agent execution (#201)
• Task prompt from file — Task prompt field can now reference an external file path instead of inline text (#157, #200)
• tool_calls grader — New grader type that validates the specific tool calls an agent makes during execution (#187, #202)

Fixed

• Webserver test resilience — Webserver tests now skip gracefully when frontend assets are not built (#204)

Changed

• Timestamped output directories — run --output-dir now groups result files by timestamp for cleaner organization (#153)
• Improved debug logging — Debug output is now more structured and useful for troubleshooting (#152)

Fixed

• --discover finds eval.yaml in nested layout — Skill discovery now correctly locates eval.yaml files in evals/{name}/ directories at the project root (#44)
• Diff grader reads post-execution workspace — The diff grader now reads files from the workspace after agent execution completes, not before (#165, #196)
• Grader config validation — Required grader configuration fields are now validated before evaluation starts (#195)
• macOS install and trigger test count — Fixed macOS binary installation and an off-by-one error in trigger test counting (#164, #184, #193)

Documentation

• Added cache command reference, prompt mode documentation, and complete YAML schema reference (#198)
• Updated demo guide and added CI/CD integration guide (#112, #89, #194)

Dependencies

• Bump defu from 6.1.4 to 6.1.6 in /site (#181)
• Bump vite from 6.4.1 to 6.4.2 in /site and /web (#182, #192)
• Bump go.opentelemetry.io/otel/sdk from 1.42.0 to 1.43.0 (#185)
• Bump astro from 5.17.3 to 5.18.1 in /site (#163)
• Bump picomatch from 4.0.3 to 4.0.4 in /site and /web (#159, #160)
• Bump smol-toml from 1.6.0 to 1.6.1 in /site (#158)

Added

• Eval coverage grid generator — New coverage output that visualizes which skills have eval coverage across grader types (#92)

Fixed

• SKILL.md injection and trigger fixture loading — waza run now correctly injects SKILL.md content into the evaluation context, loads trigger test fixtures, and passes MCP server configuration to the engine (#191)

Dependencies

• Bump h3 from 1.15.5 to 1.15.8 in /site (#144)

Added

• waza new task from-prompt command — Record Copilot sessions into task YAML files for eval creation (#110)
• Trigger heuristic grader — New grader type that scores based on trigger/anti-trigger matching heuristics (#90)
• Eval scaffolding command — waza eval new generates eval.yaml scaffolding for skills (#94)
• Multi-trial flakiness detection — Detect flaky evals across multiple trial runs (#103)
• Snapshot auto-update workflow — Diff grader can now auto-update snapshot files on mismatch (#95)
• Per-file token budget configuration — Configure token budgets per-file in .waza.yaml (#96)
• Skill-aware thresholds — waza tokens compare supports skill-specific threshold configuration (#93)
• Sensei scoring parity — WHEN triggers, spec-security, invalid level, and advisory checks 16-18 (#79)
• CI/CD integration guide — GitHub Actions and Azure DevOps integration documentation (#100)
• FileWriter service — Refactored waza init inventory with FileWriter abstraction (#63)

Fixed

• waza suggest deadlock — Execute() now applies the request timeout before calling Start(), preventing goroutine deadlock (#43)
• ResourceFile.Content type — Changed from string to []byte for proper binary file handling (#117)
• tokens compare in subdirectory — No longer shows all files as "added" when run from a subdirectory (#105)
• --output-dir ignored — Fixed --output-dir having no effect for single-skill runs (#109)
• Web dashboard build order — Build dashboard assets before Go compilation (#107)
• Test file leak — Fixed test that leaked files into the repo (#120)
• Config schema defaults — Aligned config.schema.json defaults with Go source of truth (#65)
• Skill discovery path — Discover skills under .github/skills/ directory (#69)

Changed

• Renamed config node max_workers to workers for consistency across all config types
  • This is a breaking change
• Custom YAML deserializers for config types (#106)
• Validate only known fields in YAML decoders. (#132)
• Token limits priority inverted to .waza.yaml first (#64)
• @wbreza added to CODEOWNERS (#111)
• Go 1.26+ noted in agent instruction files (#108)

Added

• Grader showcase examples demonstrating all grader types (#134)
• Reusable GitHub Actions workflow for waza evaluations (#132)
• Documentation for prompt and action_sequence grader types (#133)
• Documentation for waza dev command and compliance scoring (#131)
• Auto-loading of skills for testing (#129)
• Debug logging support (--debug flag) (#130)

Fixed

• Always output test run errors to help debug failures (#128)
• Include cwd as a skill folder when running waza (workspace fix)

Changed

• Exit codes for CI/CD integration: 0=success, 1=test failure, 2=config error (#135)
• Reordered azd-publish skill workflow steps (#127)
• Auto-merge bot registry PRs in release workflow

Added

• Skill Discovery (#3)

  • waza generate --repo <org/repo> - Scan GitHub repos for SKILL.md files
  • waza generate --scan - Scan local directory for skills
  • waza generate --all - Generate evals for all discovered skills (CI-friendly)
  • Interactive skill selection with checkboxes when not using --all

• GitHub Issue Creation (#3)

  • Post-run prompt to create GitHub issues with eval results
  • Options: create for failed tasks only, all tasks, or none
  • Issues include results table, failed task details, and suggestions
  • --no-issues flag to skip prompts (CI-friendly)

• New Modules

  • waza/scanner.py - Skill discovery from GitHub repos and local directories
  • waza/issues.py - GitHub issue creation and formatting

Changed

• Improved documentation with new feature guides
• Added skill discovery section to DEMO-SCRIPT.md
• Updated TUTORIAL.md with discovery and issue creation steps

Changed

• Renamed project from skill-eval to waza (技 - Japanese for "technique/skill")
  • New CLI command: waza (previously skill-eval)
  • New package name: waza (previously skill_eval)
  • Repository renamed to waza
• Bumped version to 0.1.0 to mark the rename milestone

Migration

If you were using skill-eval, update your scripts:

# Old
skill-eval run ./eval.yaml
pip install skill-eval

# New
waza run ./eval.yaml
pip install waza

Added

• --suggestions-file option to save improvement suggestions to markdown file
• Improved progress display with step-by-step status (tool counts, activity indicators)
• Copilot SDK usage guide in AGENTS.md

Fixed

• Fixed Copilot SDK import (from copilot import CopilotClient not copilot_sdk)
• Fixed Windows glob pattern in release workflow
• Fixed linting issues across codebase (import sorting, exception chaining, etc.)
• Clarified fixture isolation between tasks (each task gets fresh temp workspace)

Added

• CLI Commands

  • waza run - Run evaluation suites against skills
  • waza generate - Auto-generate evals from SKILL.md files
  • waza init - Initialize new eval suites interactively
  • waza report - Generate reports from results

• Eval Generation

  • Pattern-based generation from SKILL.md files
  • LLM-assisted generation with --assist flag for better tasks/fixtures
  • Support for multiple models (Claude, GPT-4, etc.)

• Executors

  • Mock executor for testing without LLM calls
  • Copilot SDK executor for real integration testing

• Graders

  • Code graders with Python assertions
  • Regex graders for pattern matching
  • LLM graders for semantic evaluation

• Features

  • Real-time progress display with conversation streaming (-v)
  • Transcript logging (--log)
  • Project context support (--context-dir)
  • LLM-powered improvement suggestions (--suggestions)

• Documentation

  • Comprehensive README with examples
  • Tutorial guide
  • Grader reference
  • Demo script for walkthroughs

Fixed

• Grader eval context now includes str, int, bool, etc.
• Transcript normalization for proper tool call detection
• YAML escaping for regex patterns with backslashes
• Progress bar now shows 100% on completion