NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #2199 most downloaded on PyPI
Framework for large language model evaluations
Latest stable
0.3.289
Last release 11 days ago
10 Nov 2026
Ships on a steady schedule
a new release about every 1 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
2 versions withdrawn
withdrawn after publishing
3 years old
266 releases · first in 2024
Drag across the ruler to compare, or tap a release to open it.
Eval Sets for running groups of tasks with automatic retries.
f1() (precision and recall in text matching) and exact() (whether normalized text matches exactly).metrics now override built in scorer metrics (previously they were merged). This enables improved re-use of existing scorers where they only change required is a different set of metrics.write_log_dir_manifest() to write a log header manifest for a log directory.store() and @subtask from solver to utils module; relocate transcript() from solver to log module.cwd that are relative paths as relative to sample working directory.--runapi flag is passed to pytest (prevents unintended token usage)chdir option from @tasks (tasks now always chdir during execution)..env files in task directories (all required vars should be specified in the global .env).strict mode for OpenAI tool calls when all function parameters are required.Your coding agent can read these notes before it upgrades. Set up the MCP server →
Store for manipulating arbitrary sample state from within solvers and tools.
Store for manipulating arbitrary sample state from within solvers and tools.Transcripts for detailed sample level tracking of model and tool calls, state changes, logging, etc.Subtasks for delegating work to helper models, sub-agents, etc.init value in default Docker compose file so that exit signals are handled correctly (substantially improves container shutdown performance).function field to ChatMessageTool to indicate the name of the function called.Support for tool calling for Llama 3.1 models on Bedrock.
strict mode in OpenAI tool calls (update to v1.40.0 of openai package required).Support for tool calling for Llama 3.1 models on Azure AI and CloudFlare.
max_tokens from 1024 to 2048.--log-images to opt back in).mistralai is now required).Fix issue affecting results of pass_at_{k} score reducer.
pass_at_{k} score reducer.Add pass_at_{k} score reducer to compute the probability of at least 1 correct sample given k epochs.
pass_at_{k} score reducer to compute the probability of at least 1 correct sample given k epochs.value_to_float string conversion (handle numbers, "true", "false", etc.)max_tokens to 4096name parameter with task created from @task decorated function.metadata available in prompt, grading, and self-critique templates.Epochs data type for specifying epochs and reducers together (deprecated epochs_reducer argument).
Epochs data type for specifying epochs and reducers together (deprecated epochs_reducer argument).INSPECT_CACHE_DIR environment variable.prompt attribute of @tool for descriptions.tool_with() function for adapting tools to have varying names and parameter descriptions.@task arguments so that dynamically created tasks can be retried.eval-retry message to terminal for filesystem based tasks.Existing symbols will continue to work but will print deprecation errors.
bootstrap_std metric with stderr for built in scorers (see rationale for details).ToolEnvironment to SandboxEnvironment and tool_environment() to sandbox() (moving the renamed types from inspect_ai.tool to inspect_ai.util). Existing symbols will continue to work but will print deprecation errors.bash(), python(), and web_search() functions from inspect_ai.solver to inspect_ai.tool. Existing symbols will continue to work but will print deprecation errors.chdir option to @task to opt-out of changing the working directory during task execution.ToolInfo parameters to be directly expressed in JSON Schema (making it much easier to pass them to model provider libraries).tool_choice="any" for OpenAI models (requires >= 1.24.0 of openai package).--parallel-tool-calls false.cwd argument to SandboxEnvironment.exec()tee rather than docker cp for Docker sandbox environment implementation of write_file().azure model arg for OpenAI provider to force binding (or not binding) to the Azure OpenAI back-end.setup field to Sample for providing a per-sample setup script.metadata field from samples.TimeoutError if a call to subprocess() or sandbox().exec() times out (formerly a textual error was returned along with a non-zero exit code).example_dataset() (and print available example dataset names).tool_error text for Anthropic tool call responses.…still work fow now, but result in a runtime deprecation warning).
eval().api_key to get_model() for explicitly specifying an API key for a model.network_mode: none for disabling networking by default in Docker tool environments.max_samples (set to 25 for the Docker provider).eval_async() (unsafe because of need to change directories for tasks). Parallel task evaluation will instead be implemented as a top-level feature of eval() and eval_async().inspect_ai.tool module (previous imports still work fow now, but result in a runtime deprecation warning).call_tools() function now operates directly on messages rather than task state.google-generativeai v0.5.3).Optional increased control over the tool use loop via the call_tools() function and new tool_calls parameter for generate().
call_tools() function and new tool_calls parameter for generate().per_epoch option for CachePolicy to allow caching to ignore epochs.choices and files when converting Sample images to base64.Various fixes for the use of Docker tool environments on Windows.
--no-toolenv-cleanup.inspect toolenv cleanup command for manually cleaning up tool environments.ToolError exception type for explicitly raising tool errors to the model. Formerly, any exception would be surfaced as a tool error to the model. Now, the ToolError exception is required for reporting to the model (otherwise other exception types go through the call stack and result in an eval error).INSPECT_LOG_DIR in .env file relative to .env file parent directory.- for delimiting --limit ranges rather than ,.Sandbox Environments for executing tool code in a sandbox.
multiple_choice() solver now has support for questions with multiple correct answers.BadRequestError (400) errors (which were formerly all treated as content moderation errors).TaskState.tools getter/setter (where the setter automatically syncs the system messages to the specified set of tools).use_tools() function now uses the TaskState.tools setter, so replaces the current set of tools entirely rather than appending to it.state.completed = False when max_messages is reached.bytes field in Logprobs and TopLogprobs.truthfulqa benchmark.intercode-ctf example.Stream samples to the evaluation log as they are completed (subject to the new --log-buffer option). Always write completed samples in the case of an
--log-buffer option). Always write completed samples in the case of an error or cancelled task."cancelled" status in eval log for tasks interrupted with SIGINT (e.g. Ctrl-C). Logs are now written for cancellations (previously they were not).--max-samples (maximum concurrent samples) to --max-connections, which will result in samples being more frequently completed and written to the log file.eval_retry(), copy previously completed samples in the log file being retried so that work is not unnecessarily repeated.inspect eval-retry command to retry a log file from a task that ended in error or cancellation.retryable_eval_logs() function and --retryable option for inspect list logs to query for tasks not yet completed within a log directory.shuffled property to datasets to determine if they were shuffled.extensions argument from list_eval_logs().Bugfix: Inspect view was not reliably updating when new evaluation logs were written.
Bugfix: results was not defined when no scorer was provided resulting in an error being thrown. Fixed by setting results = EvalResults() when no score
results was not defined when no scorer was provided resulting in an error being thrown. Fixed by setting results = EvalResults() when no scorer is provided.Update to non-beta version of Anthropic tool use (remove legacy xml tools implementation).
BREAKING: The pattern scorer has been modified to match against any (or all) regex match groups. This replaces the previous behaviour when there was m
pattern scorer has been modified to match against any (or all) regex match groups. This replaces the previous behaviour when there was more than one group, which would only match the second group.any option to indicate the model should use at least one tool (supported by Anthropic and Mistral, mapped to auto for OpenAI).list[ContentText | ContentImage].max_samples option to control how many samples are run in parallel (still defaults to running all samples in parallel).boolq.py benchmark.piqa.py benchmark.tuple return out of ToolResult type.TaskState.input_text.Remove deprecated support for matching partial model names (e.g. "gpt" or "claude").
ollama local model provider.multi_scorer() and majority_vote() functions for combining multiple scorers into a single score.model_graded_qa().TypeError for solvers and scorers not declared as async.NaN or Inf is encountered while reading log file header.Exclude null config values from listings in log viewer.
Show first log file immediately (don't wait for fetching metadata for other logs)
--version CLI arg and inspect info version command for interrogating version and runtime source path.null config values in output from inspect info log-fileFix issue with logs from S3 buckets in inspect view.
sort() method to Dataset (defaults to sorting by sample input length).write_eval_log() now ignores unserializable objects in metadata fields.
write_eval_log() now ignores unserializable objects in metadata fields.read_eval_log() now takes a str or FileInfo (for compatibility w/ list returned from list_eval_logs())..ipynb file due to lack of dependencies (e.g. nbformat).inspect view command for viewing eval log files.
inspect view command for viewing eval log files.Score now has an optional answer field, which denotes the answer text extracted from model output.ValueToFloat function for customising how textual values mapped to float.model_graded_qa more flexible with separate instruction template and grade_pattern, as well providing partial_credit as an option.chain_of_thought() and self_critique() to instruct the model to reply with ANSWER: $ANSWER at the end on its own line.match(numeric=True) (better currency and decimal handling).answer() patterns so that they detect letter and word answers both within and at the end of model output.Plan now has an optional cleanup function which can be used to free per-sample resources (e.g. Docker containers) even in the case of an evaluation error.Dataset.filter method for filtering samples using a predicate.Dataset slices (e.g. dataset[0:100]) now return a Dataset rather than list[Sample].INSPECT_LOG_DIR in .env file is now correctly resolved for execution within subdirectories.inspect list tasks and list_tasks() now only parse source files (rather than loading them), ensuring that it is fast even for task files that have non-trivial global initialisation.inspect list logs and list_eval_logs() now enumerate log files recursively by default, and only enumerate json files that match log file naming conventions.header_only option for read_eval_log() and inspect info log-file for bypassing the potentially expensive reading of samples.filter option for list_eval_logs() to filter based on log file header info (i.e. anything but samples).__main__.py entry point for invocation via python3 -m inspect_ai.ToolDef (renamed to ToolInfo).completion property on ModelOutput with no choices.- Initial release.
Nothing published for this version
Nothing published for this version
One column per month.