NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
Go modules, #1316 by repository stars
Latest stable
v0.40.1
Last release 16 days ago
06 Nov 2026
Ships fairly regularly
a new release about every 2 weeks
Nearly every release is documented
notes for 35 of 37 stable releases
Nothing withdrawn
no release was ever pulled
9 months old
61 releases · first in 2026
Drag across the ruler to compare, or tap a release to open it.
chore(deps-dev): Bump eslint from 10.7.0 to 10.8.0 in /web by @dependabot[bot] in https://github.com/microsoft/waza/pull/502
Full Changelog: https://github.com/microsoft/waza/compare/v0.38.4...v0.38.5
All notable changes to waza will be documented in this file.
The format is based on Keep a Changelog,
and this project adheres to Semantic Versioning.
tool_calls graders now use canonical tool_events[].args data, preserving MCP and custom tool arguments across results.json round trips (#474, #477).go-runewidth (#495, #497, #498, #499, #500, #502, #503).gh release create.waza update now passes the PowerShell installer URL reliably, preventing Invoke-RestMethod from failing with a null or empty Uri (#448).json_schema graders now accept the documented extract_json option in eval and task YAML, extracting exactly one JSON object or array candidate before schema validation (#457).config.mcp_servers entries now default omitted tools to ["*"], matching MCP mock behavior so configured stdio and HTTP servers expose their tools to Copilot SDK evaluations (#449).mcp_mocks now explicitly exposes declared tools to the bundled Copilot CLI, preventing valid mock servers from being skipped during Copilot SDK evaluations (#440).extract_json now rejects output containing a JSON code block plus any additional JSON document.json_schema grader can now extract exactly one JSON document from prose or a Markdown JSON code block with extract_json: true, while retaining strict pure-JSON validation by default.waza suggest now supports targeted generation with --count, --focus, --dry-run, --apply, and --force (#357, #380)checkpoints[] and on_failure policies (#358, #386)waza rubric subcommand ships in this release (#360, #381)waza spec verify to report eval coverage against SKILL.md requirements (#361, #385)waza run --otel-exporter, --otel-endpoint, --otel-headers, --otel-file, and --otel-include-payloads (#362, #383)mcp_mocks: for hermetic Copilot SDK tool-call evals (#363, #387)waza gate with stable exit codes for pass, regression, golden failure, and config errors (#364, #384)waza adversarial and eval-level adversarial: pack configuration for prompt-injection and scope-bypass checks (#365, #392)tool_events[]; tool graders can assert structured argument matchers through expect_tools[].args and tool_calls.expect[].args (#366, #388)waza run --snapshot and waza replay with a self-contained snapshot artifact format (#367, #391)schemaVersion compatibility for public artifacts (#368, #382)Last-Event-ID / lastEventId resume support for dashboard event streams, including legacy /api/events (#178, #397)waza suggest engine failures — Engine failures are now surfaced by waza suggest instead of being hidden behind success-shaped output (#330)WAZA_PROMPT_GRADER_TIMEOUT (#319)github.com/github/copilot-sdk/go to v1.0.0 and surfaced premium-request credits on the dashboard (#311)--model startup arg — The Copilot CLI validates the startup --model flag against the Copilot catalog before BYOK provider config is applied, so provider-only model IDs would fail. The --model startup arg is now skipped when a BYOK provider is configured (#305, #306)--model is now passed via CLIArgs so it correctly overrides user settings and experiment flights (#263)PATH when the bundled binary is unavailable (#300)AgentEngine cancellation handling around caller contexts to make shutdown semantics more predictable (#290)waza update command — Added an update command for upgrading local Waza installations (#288)skill_invocation graders can now assert that specific skills must not be invoked (#286, #291)waza run now respects cancellation signals more reliably (#279)waza run now reuses a shared Copilot client and auto-sizes parallel workers when --workers is unset (#135, #221)Note: This release includes the changes previously prepared under 0.32.0, which was not published.
.waza.yaml can now configure files.evalFile, files.taskGlob, and files.taskFileSuffix, with the new naming carried through scaffolding, workspace discovery, discovery mode, schemas, and docs while preserving the existing eval.yaml and tasks/*.yaml defaults (#254, closes #232)config.instruction_files and task-level instruction_files now copy files from the active context into task workspaces and append path-labeled contents to the Copilot system message (#248, closes #239)CopilotEngine instead of constructing a Copilot client directly, keeping grader execution aligned with engine configuration and preserving follow-up recovery behavior (#258, closes #54)copilot-cli bundles are updated from 1.0.2 to 1.0.49 across supported platforms, with reproducible pinned bundle generation via COPILOT_CLI_VERSION (#260, closes #244)waza new skill no longer asks for a nonstandard skill type or emits type: frontmatter, and the wizard now rejects early exits that omit required name or description fields (#261, closes #243)waza check eval discovery — Nested skills and separated evals are discovered consistently in multi-skill workspaces (#247, closes #238)SKILL.md body sections as well as frontmatter descriptions (#236, closes #223)github.com/github/copilot-sdk/go to v0.3.0, migrated session event handling to typed payloads, and refreshed transcript, logging, web API, usage collection, suggestion trace, and test coverage for the new API (#255, closes #253)go install guidance and clarified Windows/WSL install behavior (#246, closes #242; #245, closes #241).agent.md) eval support — Discover .agent.md files alongside SKILL.md, parse agent-specific frontmatter (tools, model, handoffs, mcp-servers, agents), auto-inject tool_constraint grader from agent tools: field, complete worked example under examples/custom-agent/, and new "Evaluating Custom Agents" docs guide (#226, closes #225)_output_contains expectations against file contents now work in CI without a real model. Mock response includes task metadata, file paths, and a 1KB content preview per resource (#228, closes #227)waza serve no longer crashes when stdin isn't a terminal — MCP stdio server only starts when term.IsTerminal() is true; piped input or background mode no longer kills the HTTP dashboard (#224)BenchmarkSpec → EvalSpec, TestRunner → EvalRunner. Not a breaking change for external consumers (types live in internal/) (#222).agent.md coverage to quickstart, getting-started, GUIDE, TUTORIAL, examples README; updated mock engine descriptions in INTEGRATION-TESTING and eval-yaml guide (#230)waza quality command — LLM-as-Judge skill quality scoring that evaluates skill output quality using a configurable judge model (#218)waza check now includes an advisory that flags skills with overly broad scope, helping authors tighten skill definitions (#219)--keep-workspace flag — Preserve the temporary workspace after task execution for debugging agent output (#123, #217)--no-skills flag and disabled_skills config — Disable specific skills during evaluation to isolate behavior (#126, #216)skill_directories — Specify different skill directories for individual tasks in eval YAML (#156, #215)waza models command — List all available models supported by the configured engine (#208)TestCase definitions are now properly rejected (#132, #206)output_contains_any expectation — New expectation field that passes when the agent response contains any one of the specified strings (#203)max_response_time_ms behavior rule — Enforce maximum response time constraints on agent execution (#201)prompt field can now reference an external file path instead of inline text (#157, #200)tool_calls grader — New grader type that validates the specific tool calls an agent makes during execution (#187, #202)run --output-dir now groups result files by timestamp for cleaner organization (#153)--discover finds eval.yaml in nested layout — Skill discovery now correctly locates eval.yaml files in evals/{name}/ directories at the project root (#44)Note truncated.
Your coding agent can read these notes before it upgrades. Set up the MCP server →
One column per month.
Full Changelog: v0.38.4...v0.38.5
chore(deps): bump setup-python action by @spboyer in https://github.com/microsoft/waza/pull/445
Full Changelog: https://github.com/microsoft/waza/compare/v0.38.3...v0.38.4
Full Changelog: v0.38.3...v0.38.4
All notable changes to waza will be documented in this file.
The format is based on Keep a Changelog,
and this project adheres to Semantic Versioning.
gh release create.waza update now passes the PowerShell installer URL reliably, preventing Invoke-RestMethod from failing with a null or empty Uri (#448).json_schema graders now accept the documented extract_json option in eval and task YAML, extracting exactly one JSON object or array candidate before schema validation (#457).config.mcp_servers entries now default omitted tools to ["*"], matching MCP mock behavior so configured stdio and HTTP servers expose their tools to Copilot SDK evaluations (#449).mcp_mocks now explicitly exposes declared tools to the bundled Copilot CLI, preventing valid mock servers from being skipped during Copilot SDK evaluations (#440).extract_json now rejects output containing a JSON code block plus any additional JSON document.json_schema grader can now extract exactly one JSON document from prose or a Markdown JSON code block with extract_json: true, while retaining strict pure-JSON validation by default.waza suggest now supports targeted generation with --count, --focus, --dry-run, --apply, and --force (#357, #380)checkpoints[] and on_failure policies (#358, #386)waza rubric subcommand ships in this release (#360, #381)waza spec verify to report eval coverage against SKILL.md requirements (#361, #385)waza run --otel-exporter, --otel-endpoint, --otel-headers, --otel-file, and --otel-include-payloads (#362, #383)mcp_mocks: for hermetic Copilot SDK tool-call evals (#363, #387)waza gate with stable exit codes for pass, regression, golden failure, and config errors (#364, #384)waza adversarial and eval-level adversarial: pack configuration for prompt-injection and scope-bypass checks (#365, #392)tool_events[]; tool graders can assert structured argument matchers through expect_tools[].args and tool_calls.expect[].args (#366, #388)waza run --snapshot and waza replay with a self-contained snapshot artifact format (#367, #391)schemaVersion compatibility for public artifacts (#368, #382)Last-Event-ID / lastEventId resume support for dashboard event streams, including legacy /api/events (#178, #397)waza suggest engine failures — Engine failures are now surfaced by waza suggest instead of being hidden behind success-shaped output (#330)WAZA_PROMPT_GRADER_TIMEOUT (#319)github.com/github/copilot-sdk/go to v1.0.0 and surfaced premium-request credits on the dashboard (#311)--model startup arg — The Copilot CLI validates the startup --model flag against the Copilot catalog before BYOK provider config is applied, so provider-only model IDs would fail. The --model startup arg is now skipped when a BYOK provider is configured (#305, #306)--model is now passed via CLIArgs so it correctly overrides user settings and experiment flights (#263)PATH when the bundled binary is unavailable (#300)AgentEngine cancellation handling around caller contexts to make shutdown semantics more predictable (#290)waza update command — Added an update command for upgrading local Waza installations (#288)skill_invocation graders can now assert that specific skills must not be invoked (#286, #291)waza run now respects cancellation signals more reliably (#279)waza run now reuses a shared Copilot client and auto-sizes parallel workers when --workers is unset (#135, #221)Note: This release includes the changes previously prepared under 0.32.0, which was not published.
.waza.yaml can now configure files.evalFile, files.taskGlob, and files.taskFileSuffix, with the new naming carried through scaffolding, workspace discovery, discovery mode, schemas, and docs while preserving the existing eval.yaml and tasks/*.yaml defaults (#254, closes #232)config.instruction_files and task-level instruction_files now copy files from the active context into task workspaces and append path-labeled contents to the Copilot system message (#248, closes #239)CopilotEngine instead of constructing a Copilot client directly, keeping grader execution aligned with engine configuration and preserving follow-up recovery behavior (#258, closes #54)copilot-cli bundles are updated from 1.0.2 to 1.0.49 across supported platforms, with reproducible pinned bundle generation via COPILOT_CLI_VERSION (#260, closes #244)waza new skill no longer asks for a nonstandard skill type or emits type: frontmatter, and the wizard now rejects early exits that omit required name or description fields (#261, closes #243)waza check eval discovery — Nested skills and separated evals are discovered consistently in multi-skill workspaces (#247, closes #238)SKILL.md body sections as well as frontmatter descriptions (#236, closes #223)github.com/github/copilot-sdk/go to v0.3.0, migrated session event handling to typed payloads, and refreshed transcript, logging, web API, usage collection, suggestion trace, and test coverage for the new API (#255, closes #253)go install guidance and clarified Windows/WSL install behavior (#246, closes #242; #245, closes #241).agent.md) eval support — Discover .agent.md files alongside SKILL.md, parse agent-specific frontmatter (tools, model, handoffs, mcp-servers, agents), auto-inject tool_constraint grader from agent tools: field, complete worked example under examples/custom-agent/, and new "Evaluating Custom Agents" docs guide (#226, closes #225)_output_contains expectations against file contents now work in CI without a real model. Mock response includes task metadata, file paths, and a 1KB content preview per resource (#228, closes #227)waza serve no longer crashes when stdin isn't a terminal — MCP stdio server only starts when term.IsTerminal() is true; piped input or background mode no longer kills the HTTP dashboard (#224)BenchmarkSpec → EvalSpec, TestRunner → EvalRunner. Not a breaking change for external consumers (types live in internal/) (#222).agent.md coverage to quickstart, getting-started, GUIDE, TUTORIAL, examples README; updated mock engine descriptions in INTEGRATION-TESTING and eval-yaml guide (#230)waza quality command — LLM-as-Judge skill quality scoring that evaluates skill output quality using a configurable judge model (#218)waza check now includes an advisory that flags skills with overly broad scope, helping authors tighten skill definitions (#219)--keep-workspace flag — Preserve the temporary workspace after task execution for debugging agent output (#123, #217)--no-skills flag and disabled_skills config — Disable specific skills during evaluation to isolate behavior (#126, #216)skill_directories — Specify different skill directories for individual tasks in eval YAML (#156, #215)waza models command — List all available models supported by the configured engine (#208)TestCase definitions are now properly rejected (#132, #206)output_contains_any expectation — New expectation field that passes when the agent response contains any one of the specified strings (#203)max_response_time_ms behavior rule — Enforce maximum response time constraints on agent execution (#201)prompt field can now reference an external file path instead of inline text (#157, #200)tool_calls grader — New grader type that validates the specific tool calls an agent makes during execution (#187, #202)run --output-dir now groups result files by timestamp for cleaner organization (#153)--discover finds eval.yaml in nested layout — Skill discovery now correctly locates eval.yaml files in evals/{name}/ directories at the project root (#44)waza run now correctly injects SKILL.md content into the evaluation context, loads trigger test fixtures, and passes MCP server configuration to the engine (#191)KnownFields(true) to reject unknown fields, catching typos and misconfigurations early (#132, Note truncated.
Nothing published for this version
fix: expose MCP mock tools to Copilot CLI by @spboyer in https://github.com/microsoft/waza/pull/441
Full Changelog: https://github.com/microsoft/waza/compare/v0.38.2...v0.38.3
Full Changelog: v0.38.2...v0.38.3
All notable changes to waza will be documented in this file.
The format is based on Keep a Changelog,
and this project adheres to Semantic Versioning.
mcp_mocks now explicitly exposes declared tools to the bundled Copilot CLI, preventing valid mock servers from being skipped during Copilot SDK evaluations (#440).extract_json now rejects output containing a JSON code block plus any additional JSON document.json_schema grader can now extract exactly one JSON document from prose or a Markdown JSON code block with extract_json: true, while retaining strict pure-JSON validation by default.waza suggest now supports targeted generation with --count, --focus, --dry-run, --apply, and --force (#357, #380)checkpoints[] and on_failure policies (#358, #386)waza rubric subcommand ships in this release (#360, #381)waza spec verify to report eval coverage against SKILL.md requirements (#361, #385)waza run --otel-exporter, --otel-endpoint, --otel-headers, --otel-file, and --otel-include-payloads (#362, #383)mcp_mocks: for hermetic Copilot SDK tool-call evals (#363, #387)waza gate with stable exit codes for pass, regression, golden failure, and config errors (#364, #384)waza adversarial and eval-level adversarial: pack configuration for prompt-injection and scope-bypass checks (#365, #392)tool_events[]; tool graders can assert structured argument matchers through expect_tools[].args and tool_calls.expect[].args (#366, #388)waza run --snapshot and waza replay with a self-contained snapshot artifact format (#367, #391)schemaVersion compatibility for public artifacts (#368, #382)Last-Event-ID / lastEventId resume support for dashboard event streams, including legacy /api/events (#178, #397)waza suggest engine failures — Engine failures are now surfaced by waza suggest instead of being hidden behind success-shaped output (#330)WAZA_PROMPT_GRADER_TIMEOUT (#319)github.com/github/copilot-sdk/go to v1.0.0 and surfaced premium-request credits on the dashboard (#311)--model startup arg — The Copilot CLI validates the startup --model flag against the Copilot catalog before BYOK provider config is applied, so provider-only model IDs would fail. The --model startup arg is now skipped when a BYOK provider is configured (#305, #306)--model is now passed via CLIArgs so it correctly overrides user settings and experiment flights (#263)PATH when the bundled binary is unavailable (#300)AgentEngine cancellation handling around caller contexts to make shutdown semantics more predictable (#290)waza update command — Added an update command for upgrading local Waza installations (#288)skill_invocation graders can now assert that specific skills must not be invoked (#286, #291)waza run now respects cancellation signals more reliably (#279)waza run now reuses a shared Copilot client and auto-sizes parallel workers when --workers is unset (#135, #221)Note: This release includes the changes previously prepared under 0.32.0, which was not published.
.waza.yaml can now configure files.evalFile, files.taskGlob, and files.taskFileSuffix, with the new naming carried through scaffolding, workspace discovery, discovery mode, schemas, and docs while preserving the existing eval.yaml and tasks/*.yaml defaults (#254, closes #232)config.instruction_files and task-level instruction_files now copy files from the active context into task workspaces and append path-labeled contents to the Copilot system message (#248, closes #239)CopilotEngine instead of constructing a Copilot client directly, keeping grader execution aligned with engine configuration and preserving follow-up recovery behavior (#258, closes #54)copilot-cli bundles are updated from 1.0.2 to 1.0.49 across supported platforms, with reproducible pinned bundle generation via COPILOT_CLI_VERSION (#260, closes #244)waza new skill no longer asks for a nonstandard skill type or emits type: frontmatter, and the wizard now rejects early exits that omit required name or description fields (#261, closes #243)waza check eval discovery — Nested skills and separated evals are discovered consistently in multi-skill workspaces (#247, closes #238)SKILL.md body sections as well as frontmatter descriptions (#236, closes #223)github.com/github/copilot-sdk/go to v0.3.0, migrated session event handling to typed payloads, and refreshed transcript, logging, web API, usage collection, suggestion trace, and test coverage for the new API (#255, closes #253)go install guidance and clarified Windows/WSL install behavior (#246, closes #242; #245, closes #241).agent.md) eval support — Discover .agent.md files alongside SKILL.md, parse agent-specific frontmatter (tools, model, handoffs, mcp-servers, agents), auto-inject tool_constraint grader from agent tools: field, complete worked example under examples/custom-agent/, and new "Evaluating Custom Agents" docs guide (#226, closes #225)_output_contains expectations against file contents now work in CI without a real model. Mock response includes task metadata, file paths, and a 1KB content preview per resource (#228, closes #227)waza serve no longer crashes when stdin isn't a terminal — MCP stdio server only starts when term.IsTerminal() is true; piped input or background mode no longer kills the HTTP dashboard (#224)BenchmarkSpec → EvalSpec, TestRunner → EvalRunner. Not a breaking change for external consumers (types live in internal/) (#222).agent.md coverage to quickstart, getting-started, GUIDE, TUTORIAL, examples README; updated mock engine descriptions in INTEGRATION-TESTING and eval-yaml guide (#230)waza quality command — LLM-as-Judge skill quality scoring that evaluates skill output quality using a configurable judge model (#218)waza check now includes an advisory that flags skills with overly broad scope, helping authors tighten skill definitions (#219)--keep-workspace flag — Preserve the temporary workspace after task execution for debugging agent output (#123, #217)--no-skills flag and disabled_skills config — Disable specific skills during evaluation to isolate behavior (#126, #216)skill_directories — Specify different skill directories for individual tasks in eval YAML (#156, #215)waza models command — List all available models supported by the configured engine (#208)TestCase definitions are now properly rejected (#132, #206)output_contains_any expectation — New expectation field that passes when the agent response contains any one of the specified strings (#203)max_response_time_ms behavior rule — Enforce maximum response time constraints on agent execution (#201)prompt field can now reference an external file path instead of inline text (#157, #200)tool_calls grader — New grader type that validates the specific tool calls an agent makes during execution (#187, #202)run --output-dir now groups result files by timestamp for cleaner organization (#153)--discover finds eval.yaml in nested layout — Skill discovery now correctly locates eval.yaml files in evals/{name}/ directories at the project root (#44)waza run now correctly injects SKILL.md content into the evaluation context, loads trigger test fixtures, and passes MCP server configuration to the engine (#191)KnownFields(true) to reject unknown fields, catching typos and misconfigurations early (#132, #133)max_workers renamed to workers — Config YAML key renamed for consistency across all config types (breaking change)waza check and waza tokens count now share the same counting logic for consistent results (#146)Note truncated.
JSON schema ambiguity detection — extract_json now rejects output containing a JSON code block plus any additional JSON document.
extract_json now rejects output containing a JSON code block plus any additional JSON document.JSON schema output extraction — The json_schema grader can now extract exactly one JSON document from prose or a Markdown JSON code block with extract
json_schema grader can now extract exactly one JSON document from prose or a Markdown JSON code block with extract_json: true, while retaining strict pure-JSON validation by default.chore: Release v0.37.0 registry update by @spboyer in https://github.com/microsoft/waza/pull/336
Full Changelog: https://github.com/microsoft/waza/compare/v0.37.0...v0.38.0
waza suggest now supports targeted generation with --count, --focus, --dry-run, --apply, and --force (#357, #380)checkpoints[] and on_failure policies (#358, #386)waza rubric subcommand ships in this release (#360, #381)waza spec verify to report eval coverage against SKILL.md requirements (#361, #385)waza run --otel-exporter, --otel-endpoint, --otel-headers, --otel-file, and --otel-include-payloads (#362, #383)mcp_mocks: for hermetic Copilot SDK tool-call evals (#363, #387)waza gate with stable exit codes for pass, regression, golden failure, and config errors (#364, #384)waza adversarial and eval-level adversarial: pack configuration for prompt-injection and scope-bypass checks (#365, #392)tool_events[]; tool graders can assert structured argument matchers through expect_tools[].args and tool_calls.expect[].args (#366, #388)waza run --snapshot and waza replay with a self-contained snapshot artifact format (#367, #391)schemaVersion compatibility for public artifacts (#368, #382)Last-Event-ID / lastEventId resume support for dashboard event streams, including legacy /api/events (#178, #397)Nothing published for this version
ci: Deploy Pages after release workflow by @spboyer in https://github.com/microsoft/waza/pull/265
waza run concurrency: shared Copilot client + auto-sized workers (#135) by @spboyer with @Copilot in https://github.com/microsoft/waza/pull/221waza suggest by @spboyer with @Copilot in https://github.com/microsoft/waza/pull/330Full Changelog: https://github.com/microsoft/waza/compare/v0.33.0...v0.37.0
waza suggest engine failures — Engine failures are now surfaced by waza suggest instead of being hidden behind success-shaped output (#330)Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
chore(deps): Bump astro from 6.1.8 to 6.3.2 in /site by @dependabot[bot] in https://github.com/microsoft/waza/pull/234
Full Changelog: https://github.com/microsoft/waza/compare/v0.31.0...v0.33.0
Note: This release includes the changes previously prepared under 0.32.0, which was not published.
.waza.yaml can now configure files.evalFile, files.taskGlob, and files.taskFileSuffix, with the new naming carried through scaffolding, workspace discovery, discovery mode, schemas, and docs while preserving the existing eval.yaml and tasks/*.yaml defaults (#254, closes #232)config.instruction_files and task-level instruction_files now copy files from the active context into task workspaces and append path-labeled contents to the Copilot system message (#248, closes #239)CopilotEngine instead of constructing a Copilot client directly, keeping grader execution aligned with engine configuration and preserving follow-up recovery behavior (#258, closes #54)copilot-cli bundles are updated from 1.0.2 to 1.0.49 across supported platforms, with reproducible pinned bundle generation via COPILOT_CLI_VERSION (#260, closes #244)waza new skill no longer asks for a nonstandard skill type or emits type: frontmatter, and the wizard now rejects early exits that omit required name or description fields (#261, closes #243)waza check eval discovery — Nested skills and separated evals are discovered consistently in multi-skill workspaces (#247, closes #238)SKILL.md body sections as well as frontmatter descriptions (#236, closes #223)github.com/github/copilot-sdk/go to v0.3.0, migrated session event handling to typed payloads, and refreshed transcript, logging, web API, usage collection, suggestion trace, and test coverage for the new API (#255, closes #253)go install guidance and clarified Windows/WSL install behavior (#246, closes #242; #245, closes #241)Note: This release includes the changes previously prepared under 0.32.0, which was not published.
Read more
Assets 8 Loading
There was an error while loading. Please reload this page .
All reactions
Previous 1 2 3 4 5 6 Next
Previous Next
You can’t perform that action at this time.
Nothing published for this version
refactor: complete vocabulary renames — BenchmarkSpec→EvalSpec, TestRunner→EvalRunner (#166) by @spboyer in https://github.com/microsoft/waza/pull/222
Full Changelog: https://github.com/microsoft/waza/compare/v0.30.1...v0.31.0
.agent.md) eval support — Discover .agent.md files alongside SKILL.md, parse agent-specific frontmatter (tools, model, handoffs, mcp-servers, agents), auto-inject tool_constraint grader from agent tools: field, complete worked example under examples/custom-agent/, and new "Evaluating Custom Agents" docs guide (#226, closes #225)_output_contains expectations against file contents now work in CI without a real model. Mock response includes task metadata, file paths, and a 1KB content preview per resource (#228, closes #227)waza serve no longer crashes when stdin isn't a terminal — MCP stdio server only starts when term.IsTerminal() is true; piped input or background mode no longer kills the HTTP dashboard (#224)BenchmarkSpec → EvalSpec, TestRunner → EvalRunner. Not a breaking change for external consumers (types live in internal/) (#222).agent.md coverage to quickstart, getting-started, GUIDE, TUTORIAL, examples README; updated mock engine descriptions in INTEGRATION-TESTING and eval-yaml guide (#230)README updated — Added missing waza models command documentation with usage examples and flags
waza models command documentation with usage examples and flags (#220)Full Changelog: https://github.com/microsoft/waza/compare/v0.30.0...v0.30.1
`waza quality` command (#98) — LLM-as-Judge skill quality scoring. Evaluates SKILL.md across 5 dimensions (clarity, completeness, trigger precision, s
waza quality command (#98) — LLM-as-Judge skill quality scoring. Evaluates SKILL.md across 5 dimensions (clarity, completeness, trigger precision, scope coverage, anti-patterns) using the Copilot SDK. Scored 1-5 with visual bar output. Supports --format json for CI integration. (@spboyer)
Scope reduction advisory (#183) — waza check now warns when a skill has low capability scope, detecting potential token-limit compression loss. Parses USE FOR phrases, headings, and numbered procedures as capability signals. (@diberry)
Full Changelog: https://github.com/microsoft/waza/compare/v0.29.0...v0.30.0
waza quality command — LLM-as-Judge skill quality scoring that evaluates skill output quality using a configurable judge model (#218)waza check now includes an advisory that flags skills with overly broad scope, helping authors tighten skill definitions (#219)Per-task skill directories (#156) — Tasks can now override eval-level skill_directories with their own, enabling multi-skill eval suites. (@LarryOster
Per-task skill directories (#156) — Tasks can now override eval-level skill_directories with their own, enabling multi-skill eval suites. (@LarryOsterman)
Disable skill loading (#126) — New disabled_skills config field and --no-skills CLI flag. Use disabled_skills: ["*"] to disable all skills for baseline/comparison testing. (@richardpark-msft)
Debug workspace preservation (#123) — New --keep-workspace flag preserves temp workspace directories after execution for debugging fixture and agent file issues. (@richardpark-msft)
Version update notifications (#104) — waza run now checks for new versions in the background (cached 24h, non-blocking). Disable with --no-update-check or WAZA_NO_UPDATE_CHECK=1. (@RickWinter)
Full Changelog: https://github.com/microsoft/waza/compare/v0.28.0...v0.29.0
--keep-workspace flag — Preserve the temporary workspace after task execution for debugging agent output (#123, #217)--no-skills flag and disabled_skills config — Disable specific skills during evaluation to isolate behavior (#126, #216)skill_directories — Specify different skill directories for individual tasks in eval YAML (#156, #215)Nothing published for this version
Nothing published for this version
`waza models` command (#141) — List available Copilot models with their IDs and capabilities. Requires authentication via copilot login. (@richardpark
waza models command (#141) — List available Copilot models with their IDs and capabilities. Requires authentication via copilot login. (@richardpark-msft)
Trigger test early termination (#188) — Trigger tests now cancel the agent session as soon as a skill invocation is detected, instead of waiting for the full timeout. Implemented at the execution layer via CancelOnSkillInvocation flag. (@JasonYeMSFT)
Follow-up prompts (#189) — Support multi-turn eval tasks with follow_up_prompts in task YAML. Follow-ups reuse the same session and workspace, enabling tests for conversational workflows where the agent pauses for confirmation. (@JasonYeMSFT)
Quick Start guide — New focused 5-minute quick start page on the docs site, with Mermaid workflow diagram and tabbed install options. Added as the first sidebar item.
CI integration test (#210) — Fixed the root cause of persistent ubuntu-latest CI failures. PR #203 (v0.27.0) wired up evaluateExpectations() which made output_contains checks execute for the first time — the mock executor's generic output didn't match. CI now correctly allows eval failures with mock while still catching crashes.
YAML validation audit (#132) — Verified all 10 user-facing config loaders use strict KnownFields(true) parsing. Added regression test for unknown field rejection. (@LarryOsterman)
waza models added to CLI referencefollow_up_prompts and trigger early termination documented in eval-yaml guideFull Changelog: https://github.com/microsoft/waza/compare/v0.27.0...v0.28.0
waza models command — List all available models supported by the configured engine (#208)TestCase definitions are now properly rejected (#132, #206)`tool_calls` grader (#187) — Validate which tools the agent called during execution. Supports required_tools, forbidden_tools, min_calls, and max_call
tool_calls grader (#187) — Validate which tools the agent called during execution. Supports required_tools, forbidden_tools, min_calls, and max_calls constraints with partial scoring. (@JasonYeMSFT)
output_contains_any expectation (#137) — New YAML field that passes if ANY of the listed strings appear in output (OR logic), complementing the existing output_contains (AND logic) and output_not_contains. (@LarryOsterman)
max_response_time_ms behavior rule (#136) — Enforce response time limits on eval tasks. Fails the behavior check if execution exceeds the configured threshold. (@LarryOsterman)
prompt_file for task prompts (#157) — Load task prompts from external files instead of inline YAML. Supports prompt_file: path/to/prompt.md with path traversal protection. (@LarryOsterman)
windows-latest CI failure that blocked all PRs today.runtime.GOOS for Windows compatibility.All 4 new features include updated docs:
graders.mdx) — tool_calls section addedeval-yaml.mdx) — output_contains_any, max_response_time_ms, prompt_file documentedschema.mdx) — All new fields addedFull Changelog: https://github.com/microsoft/waza/compare/v0.26.0...v0.27.0
output_contains_any expectation — New expectation field that passes when the agent response contains any one of the specified strings (#203)max_response_time_ms behavior rule — Enforce maximum response time constraints on agent execution (#201)prompt field can now reference an external file path instead of inline text (#157, #200)tool_calls grader — New grader type that validates the specific tool calls an agent makes during execution (#187, #202)suggest package: 68% → 83% — Added tests for grader docs generation, prompt templates, skill resolution, and helper functions
waza cache command documented in CLI reference (#198)mode field (independent/pairwise) added to graders guide (#198)max_attempts, group_by, fail_fast, skill_directories, required_skills, mcp_servers (#198)--output-dir now groups files by timestamp (@chlowell, #153)--discover finds eval.yaml in project-root layouts (@copilot, #44)Full Changelog: https://github.com/microsoft/waza/compare/v0.25.0...v0.26.0
run --output-dir now groups result files by timestamp for cleaner organization (#153)--discover finds eval.yaml in nested layout — Skill discovery now correctly locates eval.yaml files in evals/{name}/ directories at the project root (#44)SKILL.md content injection restored (#190) — The copilot-sdk executor now injects SKILL.md content into the model's system prompt via SystemMessage. T
SKILL.md content injection restored (#190) — The copilot-sdk executor now injects SKILL.md content into the model's system prompt via SystemMessage. The target skill's full body is injected, with compact summaries for all discovered skills. This was a regression from PR #140 which removed explicit injection.
Trigger tests now receive fixture files (#186) — Trigger test runner loads fixture files from context_dir once at init and passes them to every trigger execution request. Previously, trigger tests ran with an empty workspace.
MCP servers config now passed to copilot session (#180) — The mcp_servers section in eval YAML is now properly forwarded to the copilot SDK session, making MCP tools available to the agent during evals.
waza coverage command generates a coverage matrix showing which eval tasks exercise which graders and skill capabilities.Full Changelog: https://github.com/microsoft/waza/compare/azd-ext-microsoft-azd-waza_0.23.0...v0.25.0
waza run now correctly injects SKILL.md content into the evaluation context, loads trigger test fixtures, and passes MCP server configuration to the engine (#191)chore: bump version to 0.22.0 by @spboyer in https://github.com/microsoft/waza/pull/142
Full Changelog: https://github.com/microsoft/waza/compare/v0.22.0...v0.23.0
Nothing published for this version
Nothing published for this version
waza check uses configured token limits from .waza.yaml by @chlowell in https://github.com/microsoft/waza/pull/124
grade verb for grading run outcomes by @chlowell in https://github.com/microsoft/waza/pull/102waza run by @chlowell in https://github.com/microsoft/waza/pull/130Full Changelog: https://github.com/microsoft/waza/compare/v0.21.0...v0.22.0
Copilot SDK usage display for waza run by @chlowell in https://github.com/microsoft/waza/pull/47
--output-dir having no effect for single-skill runs by @chlowell in https://github.com/microsoft/waza/pull/109wz new task from-prompt by @richardpark-msft in https://github.com/microsoft/waza/pull/110tokens compare in subdirectory showing all files as "added" by @chlowell in https://github.com/microsoft/waza/pull/105Full Changelog: https://github.com/microsoft/waza/compare/v0.1.0...v0.21.0
waza new task from-prompt command — Record Copilot sessions into task YAML files for eval creation (#110)waza eval new generates eval.yaml scaffolding for skills (#94).waza.yaml (#96)waza tokens compare supports skill-specific threshold configuration (#93)waza init inventory with FileWriter abstraction (#63)waza suggest deadlock — Execute() now applies the request timeout before calling Start(), preventing goroutine deadlock (#43)ResourceFile.Content type — Changed from string to []byte for proper binary file handling (#117)tokens compare in subdirectory — No longer shows all files as "added" when run from a subdirectory (#105)--output-dir ignored — Fixed --output-dir having no effect for single-skill runs (#109)config.schema.json defaults with Go source of truth (#65).github/skills/ directory (#69)config node max_workers to workers for consistency across all config types
.waza.yaml first (#64)@wbreza added to CODEOWNERS (#111)Full Changelog: https://github.com/microsoft/waza/compare/v0.11.0...v0.12.0
Full Changelog: https://github.com/microsoft/waza/compare/v0.11.0...v0.12.0
Adding Microsoft SECURITY.MD by @microsoft-github-policy-service[bot] in https://github.com/microsoft/waza/pull/2
Full Changelog: https://github.com/microsoft/waza/compare/v0.10.0...v0.11.0
Nothing published for this version
Nothing published for this version
Grader showcase examples demonstrating all grader types
waza dev command and compliance scoring (#131)--debug flag) (#130)waza generate --repo - Scan GitHub repos for SKILL.md files
Skill Discovery (#3)
waza generate --repo <org/repo> - Scan GitHub repos for SKILL.md fileswaza generate --scan - Scan local directory for skillswaza generate --all - Generate evals for all discovered skills (CI-friendly)--allGitHub Issue Creation (#3)
--no-issues flag to skip prompts (CI-friendly)New Modules
waza/scanner.py - Skill discovery from GitHub repos and local directorieswaza/issues.py - GitHub issue creation and formattingNothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Renamed project from `skill-eval` to `waza` (技 - Japanese for "technique/skill")
skill-eval to waza (技 - Japanese for "technique/skill")
waza (previously skill-eval)waza (previously skill_eval)wazaIf you were using skill-eval, update your scripts:
# Old
skill-eval run ./eval.yaml
pip install skill-eval
# New
waza run ./eval.yaml
pip install waza
--suggestions-file option to save improvement suggestions to markdown file
--suggestions-file option to save improvement suggestions to markdown filefrom copilot import CopilotClient not copilot_sdk)waza run - Run evaluation suites against skills
CLI Commands
waza run - Run evaluation suites against skillswaza generate - Auto-generate evals from SKILL.md fileswaza init - Initialize new eval suites interactivelywaza report - Generate reports from resultsEval Generation
--assist flag for better tasks/fixturesExecutors
Graders
Features
-v)--log)--context-dir)--suggestions)Documentation
str, int, bool, etc.