github.com/microsoft/waza

Added

• Custom agent (.agent.md) eval support — Discover .agent.md files alongside SKILL.md, parse agent-specific frontmatter (tools, model, handoffs, mcp-servers, agents), auto-inject tool_constraint grader from agent tools: field, complete worked example under examples/custom-agent/, and new "Evaluating Custom Agents" docs guide (#226, closes #225)

Fixed

• Mock engine echoes file content — _output_contains expectations against file contents now work in CI without a real model. Mock response includes task metadata, file paths, and a 1KB content preview per resource (#228, closes #227)
• waza serve no longer crashes when stdin isn't a terminal — MCP stdio server only starts when term.IsTerminal() is true; piped input or background mode no longer kills the HTTP dashboard (#224)

Changed

• Vocabulary renames — Internal types renamed: BenchmarkSpec → EvalSpec, TestRunner → EvalRunner. Not a breaking change for external consumers (types live in internal/) (#222)

Documentation

• Cross-reference audit for recent renames + custom agent feature: added .agent.md coverage to quickstart, getting-started, GUIDE, TUTORIAL, examples README; updated mock engine descriptions in INTEGRATION-TESTING and eval-yaml guide (#230)

Dependencies

• Bump postcss from 8.5.6 to 8.5.12 in /site (#229)

Documentation

• Updated README with missing CLI commands — Added documentation for recently-added CLI commands that were missing from the README (#220)

Added

• waza quality command — LLM-as-Judge skill quality scoring that evaluates skill output quality using a configurable judge model (#218)
• Scope-reduction advisory check — waza check now includes an advisory that flags skills with overly broad scope, helping authors tighten skill definitions (#219)

Added

• --keep-workspace flag — Preserve the temporary workspace after task execution for debugging agent output (#123, #217)
• --no-skills flag and disabled_skills config — Disable specific skills during evaluation to isolate behavior (#126, #216)
• Non-blocking version update check — CLI now checks for newer waza versions in the background without slowing startup (#104, #214)
• Per-task skill_directories — Specify different skill directories for individual tasks in eval YAML (#156, #215)

Dependencies

• Bump astro and @astrojs/starlight in /site (#212)

Added

• Follow-up prompts in eval YAML — Tasks can now include pre-written follow-up prompts for multi-turn evaluation conversations (#189, #209)
• waza models command — List all available models supported by the configured engine (#208)
• Early termination for trigger tests — Trigger tests can now stop early once the target skill is invoked, reducing evaluation time (#207)

Fixed

• Stricter YAML validation — Audited all YAML parsers; unknown fields in TestCase definitions are now properly rejected (#132, #206)
• Test fixture assertion syntax — Fixed invalid Python expression in a test fixture assertion (#197)
• CI integration test stability — CI integration tests now correctly handle expected eval failures when using the mock executor (#210)

Documentation

• Added Quick Start guide to the documentation site (#205)

Added

• output_contains_any expectation — New expectation field that passes when the agent response contains any one of the specified strings (#203)
• max_response_time_ms behavior rule — Enforce maximum response time constraints on agent execution (#201)
• Task prompt from file — Task prompt field can now reference an external file path instead of inline text (#157, #200)
• tool_calls grader — New grader type that validates the specific tool calls an agent makes during execution (#187, #202)

Fixed

• Webserver test resilience — Webserver tests now skip gracefully when frontend assets are not built (#204)

Changed

• Timestamped output directories — run --output-dir now groups result files by timestamp for cleaner organization (#153)
• Improved debug logging — Debug output is now more structured and useful for troubleshooting (#152)

Fixed

• --discover finds eval.yaml in nested layout — Skill discovery now correctly locates eval.yaml files in evals/{name}/ directories at the project root (#44)
• Diff grader reads post-execution workspace — The diff grader now reads files from the workspace after agent execution completes, not before (#165, #196)
• Grader config validation — Required grader configuration fields are now validated before evaluation starts (#195)
• macOS install and trigger test count — Fixed macOS binary installation and an off-by-one error in trigger test counting (#164, #184, #193)

Documentation

• Added cache command reference, prompt mode documentation, and complete YAML schema reference (#198)
• Updated demo guide and added CI/CD integration guide (#112, #89, #194)

Dependencies

• Bump defu from 6.1.4 to 6.1.6 in /site (#181)
• Bump vite from 6.4.1 to 6.4.2 in /site and /web (#182, #192)
• Bump go.opentelemetry.io/otel/sdk from 1.42.0 to 1.43.0 (#185)
• Bump astro from 5.17.3 to 5.18.1 in /site (#163)
• Bump picomatch from 4.0.3 to 4.0.4 in /site and /web (#159, #160)
• Bump smol-toml from 1.6.0 to 1.6.1 in /site (#158)

Added

• Eval coverage grid generator — New coverage output that visualizes which skills have eval coverage across grader types (#92)

Fixed

• SKILL.md injection and trigger fixture loading — waza run now correctly injects SKILL.md content into the evaluation context, loads trigger test fixtures, and passes MCP server configuration to the engine (#191)

Dependencies

• Bump h3 from 1.15.5 to 1.15.8 in /site (#144)

Added

• waza new task from-prompt command — Record Copilot sessions into task YAML files for eval creation (#110)
• Trigger heuristic grader — New grader type that scores based on trigger/anti-trigger matching heuristics (#90)
• Eval scaffolding command — waza eval new generates eval.yaml scaffolding for skills (#94)
• Multi-trial flakiness detection — Detect flaky evals across multiple trial runs (#103)
• Snapshot auto-update workflow — Diff grader can now auto-update snapshot files on mismatch (#95)
• Per-file token budget configuration — Configure token budgets per-file in .waza.yaml (#96)
• Skill-aware thresholds — waza tokens compare supports skill-specific threshold configuration (#93)
• Sensei scoring parity — WHEN triggers, spec-security, invalid level, and advisory checks 16-18 (#79)
• CI/CD integration guide — GitHub Actions and Azure DevOps integration documentation (#100)
• FileWriter service — Refactored waza init inventory with FileWriter abstraction (#63)

Fixed

• waza suggest deadlock — Execute() now applies the request timeout before calling Start(), preventing goroutine deadlock (#43)
• ResourceFile.Content type — Changed from string to []byte for proper binary file handling (#117)
• tokens compare in subdirectory — No longer shows all files as "added" when run from a subdirectory (#105)
• --output-dir ignored — Fixed --output-dir having no effect for single-skill runs (#109)
• Web dashboard build order — Build dashboard assets before Go compilation (#107)
• Test file leak — Fixed test that leaked files into the repo (#120)
• Config schema defaults — Aligned config.schema.json defaults with Go source of truth (#65)
• Skill discovery path — Discover skills under .github/skills/ directory (#69)

Changed

• Renamed config node max_workers to workers for consistency across all config types
  • This is a breaking change
• Custom YAML deserializers for config types (#106)
• Validate only known fields in YAML decoders. (#132)
• Token limits priority inverted to .waza.yaml first (#64)
• @wbreza added to CODEOWNERS (#111)
• Go 1.26+ noted in agent instruction files (#108)

Added

• Grader showcase examples demonstrating all grader types (#134)
• Reusable GitHub Actions workflow for waza evaluations (#132)
• Documentation for prompt and action_sequence grader types (#133)
• Documentation for waza dev command and compliance scoring (#131)
• Auto-loading of skills for testing (#129)
• Debug logging support (--debug flag) (#130)

Fixed

• Always output test run errors to help debug failures (#128)
• Include cwd as a skill folder when running waza (workspace fix)

Changed

• Exit codes for CI/CD integration: 0=success, 1=test failure, 2=config error (#135)
• Reordered azd-publish skill workflow steps (#127)
• Auto-merge bot registry PRs in release workflow

Added

• Skill Discovery (#3)

  • waza generate --repo <org/repo> - Scan GitHub repos for SKILL.md files
  • waza generate --scan - Scan local directory for skills
  • waza generate --all - Generate evals for all discovered skills (CI-friendly)
  • Interactive skill selection with checkboxes when not using --all

• GitHub Issue Creation (#3)

  • Post-run prompt to create GitHub issues with eval results
  • Options: create for failed tasks only, all tasks, or none
  • Issues include results table, failed task details, and suggestions
  • --no-issues flag to skip prompts (CI-friendly)

• New Modules

  • waza/scanner.py - Skill discovery from GitHub repos and local directories
  • waza/issues.py - GitHub issue creation and formatting

Changed

• Improved documentation with new feature guides
• Added skill discovery section to DEMO-SCRIPT.md
• Updated TUTORIAL.md with discovery and issue creation steps

Changed

• Renamed project from skill-eval to waza (技 - Japanese for "technique/skill")
  • New CLI command: waza (previously skill-eval)
  • New package name: waza (previously skill_eval)
  • Repository renamed to waza
• Bumped version to 0.1.0 to mark the rename milestone

Migration

If you were using skill-eval, update your scripts:

# Old
skill-eval run ./eval.yaml
pip install skill-eval

# New
waza run ./eval.yaml
pip install waza

Added

• --suggestions-file option to save improvement suggestions to markdown file
• Improved progress display with step-by-step status (tool counts, activity indicators)
• Copilot SDK usage guide in AGENTS.md

Fixed

• Fixed Copilot SDK import (from copilot import CopilotClient not copilot_sdk)
• Fixed Windows glob pattern in release workflow
• Fixed linting issues across codebase (import sorting, exception chaining, etc.)
• Clarified fixture isolation between tasks (each task gets fresh temp workspace)

Added

• CLI Commands

  • waza run - Run evaluation suites against skills
  • waza generate - Auto-generate evals from SKILL.md files
  • waza init - Initialize new eval suites interactively
  • waza report - Generate reports from results

• Eval Generation

  • Pattern-based generation from SKILL.md files
  • LLM-assisted generation with --assist flag for better tasks/fixtures
  • Support for multiple models (Claude, GPT-4, etc.)

• Executors

  • Mock executor for testing without LLM calls
  • Copilot SDK executor for real integration testing

• Graders

  • Code graders with Python assertions
  • Regex graders for pattern matching
  • LLM graders for semantic evaluation

• Features

  • Real-time progress display with conversation streaming (-v)
  • Transcript logging (--log)
  • Project context support (--context-dir)
  • LLM-powered improvement suggestions (--suggestions)

• Documentation

  • Comprehensive README with examples
  • Tutorial guide
  • Grader reference
  • Demo script for walkthroughs

Fixed

• Grader eval context now includes str, int, bool, etc.
• Transcript normalization for proper tool call detection
• YAML escaping for regex patterns with backslashes
• Progress bar now shows 100% on completion