NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #2199 most downloaded on PyPI
Framework for large language model evaluations
Latest stable
0.3.289
Last release 8 days ago
10 Nov 2026
Ships on a steady schedule
a new release about every 1 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
2 versions withdrawn
withdrawn after publishing
3 years old
266 releases · first in 2024
Drag across the ruler to compare, or tap a release to open it.
Open AI: Treat UnprocessableEntityError as bad request so we can include the request payload in the error message.
UnprocessableEntityError as bad request so we can include the request payload in the error message.Remove support for goodfire model provider (dependency conflicts).
goodfire model provider (dependency conflicts).description without name.Bugfix: Suppress link click behavior in vscode links.
Your coding agent can read these notes before it upgrades. Set up the MCP server →
Model API: New execute_tools() function (replaces deprecated call_tools() function) which handles agent handoffs that occur during tool calling.
submit_append option to append the submit tool output to the completion rather than replacing the completion (note that the new react() agent appends by default).call_tools() function) which handles agent handoffs that occur during tool calling.generate_loop() method for calling generate with a tool use loop.Model (works only with providers that don't require an async close).tool_choice="none" (added in v0.49.0, which is now required).logprobs to pass 1 rather than True (protocol change).bash_session() and web_browser() now create a distinct sandbox process each time they are instantiated.openai/computer-use-preview)task_with() and tool_with() no longer copy the input task or tool (rather, they modify it in place and return it).name (save registry_name separately).instance option for store_as() for using multiple instances of a StoreModel within a sample.StoreModel within another StoreModel.sandbox_default() context manager for temporarily changing the default sandbox.write_file() function now gracefully handles larger input file sizes (was failing on files > 2MB).user and assistant solver events.score() function within Jupyter notebooks.eval() when a cancellation occurs.api_version model argument for OpenAI on Azure.None passed to tool call by model for optional parameters..compose.yml when not in working directory.Bugfix: Correct handling of backward compatibility for inspect-web-browser-tool image.
max_tasks is greater than total tasksRequirements: Temporarily upper-bound rich to < 14.0.0 to workaround issue.
rich to < 14.0.0 to workaround issue.Google: Compatibility with httpx client in google-genai >= 1.8.0 (which is now required).
google-genai >= 1.8.0 (which is now required).mistralai >= v1.6.0 (which is now required).Google: Compatibility with v1.7 of google-genai package (create client per-generate request)
OpenAI: Ensure that assistant messages always have the msg_ prefix in responses API.
msg_ prefix in responses API.New think() tool that provides models with the ability to include an additional thinking step.
openai/azure/gpt-4o-mini or mistral/azure/Mistral-Large-2411).--metadata option to eval for associating metadata with eval runs.think.bash_session() tool for creating a stateful bash shell that retains its state across calls from the model.
reasoning_effort parameter to o1-preview (as it is not supported).Model API: Specifying a default model (e.g. --model) is no longer required (as some evals have no model or use get_model() for model access).
--model) is no longer required (as some evals have no model or use get_model() for model access).model, and model is no longer a required axis for parallel tasks.id for ChatMessage when deserialising (id is now str | None and is only populated when messages are directly created).reasoning_tokens for standard thinking blocks (redacted thinking not counted).APIError status codes for retry.--env option for defining environment variables for the duration of the inspect process.inspect view and viewing the log in a browser.store_as()) from log.solver list to eval() (decorate chain function with @solver).Bugfix: Exclude chat message id from cache key (fixes regression in model output caching).
id from cache key (fixes regression in model output caching).The ModelAPI class now has a should_retry() method that replaces the deprecated is_rate_limit() method.
ModelAPI class now has a should_retry() method that replaces the deprecated is_rate_limit() method.generate().inspect trace http command which will show all HTTP requests for a run.max_retries and timeout configuration options. These options now exclusively control Inspect's outer retry handler; model providers use their default behaviour for the inner request, which is typically 2-4 retries and a service-appropriate timeout.tool_choice option.ChatMessage now includes an id field (defaults to auto-generated uuid).mistralai is now required).@task(chdir=True) to preserve the previous behavior.parallel_tool_calls correctly for OpenAI models served through Azure.Computer: Updated tool definition to match improvements in Claude Sonnet 3.7.
Anthropic: Support for extended thinking features of Claude Sonnet 3.7 (minimum version of anthropic package bumped to 0.47.1).
anthropic package bumped to 0.47.1).ContentReasoning type for representing model reasoning blocks.reasoning_tokens for setting maximum reasoning tokens (currently only supported by Claude Sonnet 3.7)reasoning_history can now be specified as "none", "all", "last", or "auto" (which yields a provider specific recommended default).reasoning_tokens when reported.None for assistant content (can happen when there is a refusal).working_start attribute to events and completed and working_time to model, tool, and subtask events.task quit command for giving up on tasks.TimeoutError for running shell commands in the computer tool container.working_limit option for specifying a maximum working time (e.g. model generation, tool calls, etc.) for samples.
SandboxEvent to transcript for recording sandbox execution and I/O.as_type() function for checked downcasting of SandboxEnvironmentstate.completed=True when entering scoring (basic_agent() no longer sets completed so can be used in longer compositions of solvers).uuid property to TaskState and EvalSample (globally unique identifier for sample run).cleanup to tasks for executing a function at the end of each sample run.bridge() is now compatible with the use of a custom OPENAI_BASE_URL.mistralai package to 1.5 (required for working_limit).fsspec version constraints to avoid pip errors when installing alongside datasets.dict as a parameter.sample_id for --no-sandbox-cleanup.Google provider updated to use the Google Gen AI SDK, which is now the recommended API for Gemini 2.0 models.
compose up when no services startup successfully.--no-sandbox-cleanupTimeoutError for subprocess timeouts (ensure kill/cleanup of timed out process).Task display: Improve spacing/layout of final task display.
Memoize calls to get_model() so that model instances with the same parameters are cached and re-used (pass memoize=False to disable).
get_model() so that model instances with the same parameters are cached and re-used (pass memoize=False to disable).Model class for optional scoped usage of model clients.assistant_message() solver.prompt_template(), and system/user/assistant solvers.Docker: Correct compose file generation for Dockerfiles w/ custom stem or extension.
Compatibility with textual 2.0 (which had several breaking changes).
null values.Metrics now take list[SampleScore] rather than list[Score] (previous signature is deprecated but still works with a warning).
cluster parameter for the stderr() metric.inspect score command and score() function).list[SampleScore] rather than list[Score] (previous signature is deprecated but still works with a warning).var() metric.sandbox argument for running in non-default sandboxes.ScoreEvent (with intermediate=True) when the score() function is called.source field to InfoEvent and use it for events logged by the human agent..Dockerfile extension.container_name (incompatible with epochs > 1).compose up timeout when there are healthcheck entries for services.log_dir is writeable at startup.None).Add shuffle_choices to dataset and dataset loading functions. Deprecate shuffle parameter to the multiple_choice solver.
shuffle_choices to dataset and dataset loading functions. Deprecate shuffle parameter to the multiple_choice solver.stop_words param to the f1 scorer. stop_words will be removed from the target and answer during normalization.inspect_ai.tool.beta into inspect_ai.tool).tee for write_file operations.type parameter of answer() to pattern to address registry serialisation error.Various improvements for reasoning models including extracting reasoning content from assistant messages.
reasoning_effort, max_tokens, temperature, and parallel_tool_calls correctly for o3 models.content_filter stop reason.model_length StopReason.Computer: Enable viewing computer tool's remote mouse cursor via VNC.
SampleLimitExceededError with current state so that messages, etc. are preserved when limits are hit.incorrect_message can now optionally be an async function.suffix from eval-set CLI args.Exception from sandboxenv_init (allow cancelled to propagate)Agent Bridge for integrating external agent frameworks with Inspect.
@wraps to functions wrapped by Inspect decorators to preserve type information.refusal field from assistant message when provided.openai/azure)anthropic/bedrock, anthropic/vertex).extra_body model arg (for adding additional JSON properties to the request)tools to state so that tools added in init are preserved.Beta version of computer() tool which models with a computer desktop environment.
user_message() solver for appending parameterised user messages.prompt_template(), system_message() and user_message() solver now also include the sample store in substitution parameters.state.completed for limit enforcement).SampleLimitExceededError.model_input option that determines how tool call result content is played back to the model.ToolDef with **kwargs: Any (just pass them through).fail_on_error option)plain mode with periodic updates on progress, metrics, etc.Literal["a", "b", "c"])) in tool function declarations.Print model conversations to terminal with --display=conversation (was formerly --trace, which is now deprecated).
timeout_retry option (defaulting to True) to exec() function.type and optional container properties to SandboxConnection.task_with() function for creating task variants.--filter argument to trace CLI commands for filtering on trace log message content.--display=conversation (was formerly --trace, which is now deprecated).bootstrap_std metric to bootstrap_stderr (deprecate bootstrap_std)Tracing API for custom trace logging.
StopReason)max_completion_tokens option for o1 full.error and web_at fields are present in response.Store from existing dictionary.metadata_as and store_as typed accessors for sample metadata and store.None are now supported.Human Agent solver for human baselining of computing tasks.
Sample store and metadata using Pydantic models.Task level (eval level approval policies take precedence).ContentText and ContentImage.SandboxConnection that contains login information from sandboxes.display_type() function for detecting the current display type (e.g. "full", "rich", etc.)eval() running in multiple processes at once (trace file per-process)docker build and docker pull commands.store.get() not auto-inserting default value.Bedrock: redact authentication model args from eval logs.
temperature is used with o1 models (as it is not supported).Tracing for diagnosing runs with unterminated action (e.g. model calls, docker commands, etc.).
--no-score-display option to disable realtime scoring metrics.logprobs.OpenAI: Support for o1 including native tool calling and reasoning_effort generation option.
reasoning_effort generation option.setup step that always runs even if solver is replaced.model_args passed through to session.Client.jpeg images.max_sandboxes option for (per-provider) maximum number of running sandboxes.max_tasks as a lower bound for max_samples.eval format logs.--sample-id integer comparisons.None)Eval: --sample-id option for evaluating specific sample id(s).
--sample-id option for evaluating specific sample id(s).emulate_tools model arg to force tool emulation (emulation is enabled by default for Llama models).max_tool_output parameter to override default max tool output from generate config.--trace mode.Bugfix: Task display fails to load when no scorers are defined for a task.
Tools: Improved typing/schema support (unions, optional params, enums).
append argument to use_tools() for adding (rather than replacing) the currently available tools.BaseModel types for sandbox config (formerly only a file path could be passed).Enter key).input_panel() API for adding custom panels to the fullscreen task display..eval log reading performance for remote filesystem (eagerly fetch log to local buffer).token_usage property to TaskState which has current total tokens used across all calls to generate() (same value that is used for enforcing token limits).time field to ModelOutput that records total time spent within call to ModelAPI generate().tokenizer_call_args dict to specify custom args during tokenization, such as max_length and truncation.None for content.hf_dataset now explicitly requires the split argument (previously, it would crash when not specified).Logging: Only call CreateBucket on Amazon S3 when the bucket does not already exist.
Realtime display of sample transcripts (including ability to cancel running samples).
EvalLog now includes a location property indicating where it was read from.max_samples across sandbox and non-sandbox evals (both now apply max_samples per task, formerly evals with sandboxes applied max_samples globally).--login option so that e.g. .bashrc is read before executing the command.logprobs and other new 1.5 (002 series) options.cached argument to control whether to use a previously cached version of the dataset if available (defaults to True).revision option to load a specific branch or commit SHA (when using revision datasets are always revalidated on Hugging Face, i.e. cached is ignored).Basic agent: Ensure that the scorer is only run once when max_attempts = 1.
eval is now the default log format (use --log-format=json to use old format).
--log-format=json to use old format).--no-log-images).eval format log files.inspect log convert to convert a single log file.exec() output limit from 1 MiB to 10 MiB.time_limit option for specifying a maximum execution time for samples.
.eval log files.INSPECT_DISABLE_MODEL_API into generate() (as opposed to get_model()).eval files as logs (don't apply file name pattern restrictions as we do with .json).metadata field to ModelOutput and provide various fields for the Groq provider.Revert change to single epoch reducer behavior (regressed some scoring scenarios).
New binary log format which yields substantial size and speed improvements (JSON format log files are still fully supported and utilities for converti
--model-config, --task-config, and --solver-config CLI arguments for specifying model, task, and solver args using a JSON or YAML config file.casefold() for case-insensitive compare in includes(), match(), exact(), and f1() scorers.strict tool calling (sporadically supported across models and we already internally validate).read_eval_log_sample() for JSON log files.SandboxEnvironment.exec() output streams to 1 MiB. Limit SandboxEnvironment.read_file() to 100 MiB.INSPECT_DISABLE_MODEL_API environment variable for disabling all Model APIs save for mockllm.tool_call_id param to ModelOutput.for_tool_call().file_dataset() function.Open log files in binary mode when reading headers (fixes ijson deprecation warning).
--tags option to eval for tagging evaluation runs.eval().include_history option to model graded scorers to optionally include the full chat history in the presented question.delimiter option to csv_dataset() (defaults to ",")list and dict of registry objects when logging plan.model_usage field to EvalSample to record token usage by model for each sample.…used per sample ( message_limit replaces deprecated max_messages).
as_dict() utility method to Scoretoken_limit and message_limit) for capping the number of tokens or messages used per sample ( message_limit replaces deprecated max_messages).metadata field to Task and record in log EvalSpec.--log-level-transcript option for separate control of log entries recorded in the eval log filefail_on_error option for eval_retry() and inspect eval-retry.self_critique() and model_graded_qa().--port and --host on Inspect View.Add interactive option to web_browser() for disabling interactive tools (clicking, typing, and submitting forms).
interactive option to web_browser() for disabling interactive tools (clicking, typing, and submitting forms).basic_agent(), defer to task max_messages if none is specified for the agent (default to 50 is the task does not specify max_messages).content parameter to ModelOutput.for_tool_call().sample_reductions when returning eval logs with header_only=True.The sample transcript will now display the target for scoring in the Score Event (for newly run evaluations).
max_messages on TaskState.max_messages option for basic_agent() (defaulting to 50) and use it rather than any task max_messages defined.Rename web_browser_tools() to web_browser(), and don't export individual web browsing tools.
web_browser_tools() to web_browser(), and don't export individual web browsing tools.parallel option to @tool decorator and specify parallel=False for web browsing tools.</tool_call> end sequence for Llama models.Move evals into inspect_evals package.
Web Browser tool which provides a headless Chromium browser that supports navigation, history, and mouse/keyboard interactions.
auto_id option for dataset readers to assign an auto-incrementing ID to records.Catch o1-preview "invalid_prompt" exception and convert to normal content_filter refusal.
metadata field in the first epoch@task.Support for max_tokens on OpenAI o1 models (map to max_completion_tokens).
max_tokens on OpenAI o1 models (map to max_completion_tokens).inspect viewepochs is less than 1StopReason: Added "model_length" for exceeding token window and renamed "length" to "max_tokens".
fork().--no-ansi or INSPECT_NO_ANSImultiple_choice() and export MultipleChoiceTemplate enumerationx-default to be referred to by their declared service name.max_messages for use of basic_agent() (as without it, the agent could end up in an infinite loop).answer, explanation, and metadata only if equal across all epochs.Fix issue w/ subtasks not getting a fresh store() (regression from introduction of fork() in v0.3.30)
fork() in v0.3.30)web_search() provider environment variables.Deprecated Plan in favor of Solver (with chain() function to compose multiple solvers).
Plan in favor of Solver (with chain() function to compose multiple solvers).max_tool_output generation option (defaults to 16KB).header_only log reading (switch from json-stream to ijson).eval-set (run a single eval then stop).epochs in task status (formerly wasn't included for multiple task display)info() strings as markdownAdded fork() function to fork a TaskState and evaluate it against multiple solvers in parallel.
fork() function to fork a TaskState and evaluate it against multiple solvers in parallel.answer, explanation, and metadata.inspect info log-typescache argument to basic_agent() for specifying cache policy for the agent.cache field to ModelEvent to track cache reads and writes.Added --plan and -P arguments to eval and eval-set commands for replacing the task default plan with another one.
--plan and -P arguments to eval and eval-set commands for replacing the task default plan with another one.eval() or eval_set() with a plan argument.--log-images).epoch_reducer from being used in eval-retry.epoch from overriding task level epoch.basic_agent() that provides a ReAct tool loop with support for retries and encouraging the model to continue if its gives up or gets stuck.
system_message() now supports custom parameters and interpolation of metadata values from Sample.generate() solver now accepts arbitrary generation config params.use_tools() now accepts a variadic list of Tool in addition to literal list[Tool].bash() and python() tools now have a user parameter for choosing an alternate user to run code as.bash() and python() tools now always return stderr and stdout no matter the exit status.debug_errors option to eval() to raise task errors (rather than logging them) so they can be debugged.eval() from a script or notebook.eval_set() issue with cleaning up failed logs on S3.Fix missing timestamp issue with running eval_set() with an S3-backed log directory.
eval_set() with an S3-backed log directory.f1() and exact() scorers.exact() scorer.One column per month.