NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #2199 most downloaded on PyPI
Framework for large language model evaluations
Latest stable
0.3.289
Last release 7 days ago
10 Nov 2026
Ships on a steady schedule
a new release about every 1 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
2 versions withdrawn
withdrawn after publishing
3 years old
266 releases · first in 2024
Drag across the ruler to compare, or tap a release to open it.
Bugfix: Fix Google provider serialization of thought signatures on replay.
Added cache configuration to GenerateConfig (formerly was only available as a parameter to generate()).
cache configuration to GenerateConfig (formerly was only available as a parameter to generate()).on_continue can now return a new AgentState.APIConnectionError.reasoning_effort="none" (now available with gpt-5.1).None as value of arguments when parsing tool calls.Your coding agent can read these notes before it upgrades. Set up the MCP server →
OpenAI: Show reasoning summaries by default (auto-detect whether current account is capable of reasoning summaries and fallback as required).
logprobs and top_logprobs in Responses API (note that logprobs are not supported for reasoning models).xai_sdk package (rather than using OpenAI compatible endpoint).web_search() tool.reasoning_enabled model arg to optionally disable reasoning for hybrid models.eval_set_idcontent of type str in Responses API agent bridge.Eval Set: Task identifiers can now vary on GenerateConfig and solver (which enables sweeping over these variables).
GenerateConfig and solver (which enables sweeping over these variables).ModelEvent.call by default, which prevents O(N) memory footprint for reading transcripts.ResponseOutputText in v2.7.0 of openai package.reasoning_effort parameter (only supported by grok-3-mini and only low and highvalues are supported).Google: Correct capture and playback of thought_signature in ContentReasoning blocks.
thought_signature in ContentReasoning blocks.@scanner functions as scorers.output during execution (only condense call).logs in data frame functions.Google: Distribute citations from web search to individual ContentText parts (rather than concatenating into a single part).
Tests: Skip git revision detection and realtime logging during pytest runs to improve test performance.
OpenAI: Handle Message input types that have no "type" field in responses API.
Message input types that have no "type" field in responses API.override_api_key() hook.dict as well as list for underlying data.edit_score() silently editing only first epoch in multi-epoch evaluations (now requires explicit epoch parameter).Added model API for Hugging Face Inference Providers.
messages_to_openai_responses() function.gpt-5-pro by default.parallel_tool_calls option for tool choice.logprobs and top_logprobs.model_length stop reason based on additional error pattern.content can never be empty.edit_score() and recompute_metrics() functions for modifying evaluation scores with provenance tracking and metric recomputation.attempt_timeout to GenerateConfig (governs timeout for individual attempts and still retries if timeout is exceeded).ModelEvent immediately to prevent O(N) memory usage for long message histories.inspect view server using uvicorn / fastapi.Union for TypeAlias (required by Python 3.14).umask that led to permission problems with bash_session tool.list and dict parameters (rather use None and initialize on use).SubtaskEvent.input values that aren't of the required dict type.OpenAI: Support for tool calls returning images (requires v2.0 of openai package, which is now required).
openai package, which is now required).run() function.inspect_ai.event module with event related tyeps and functions.rich > 13.3.3 save for 14.0.0 (which had an infinite recursion bug affecting stack traces with exception groups).resolve_attachments in score command when stream=True.fail_on_error=False.Google: Manage Google client lifetime to scope of call to generate().
generate().inspect score command (and inspect_score function) when scoring log files on S3.OpenAI: Capture reasoning summaries even when there is encrypted reasoning content.
Bugfix: Update various tests to react to Google's deprecation of old models.
retry_refusals loop.detail along with images.--num-choices is specified.SampleScores column group for extracting score answer, explanation, and metadata.inspect-ai package installation type detection code.update_plan and shell tool inputs.EvalSet when optional values are missing.Sandbox tools: bash_session, text_editor, and sandbox MCP servers no longer require a separate pipx install (they are now automatically injected into
retry_refusals option for automatically retrying refusals a set number of times.convert_eval_logs().convert_eval_logs().model_graded_qa(), model_graded_fact()) now look for the "grader" model-role by default.on_sample_scoring() and on_model_cache_usage() hooks.on_run_end() even when the eval is cancelled.None to indicate that they did not score the sample. Such samples are excluded from reductions and metrics.Sequence and Mapping types for metrics on scorer decorator.async_score in eval score.Anthropic: Support for images with mime type image/bmp.
store=False from bridge client and don't insist on id being included with reasoning (as it is not returned in store=False mode).attachments:/ appearing in content when viewing running samples.OpenAI: Correct serialization of web search tool calls (prevent 400 errors).
Agent Bridge: Option to force the sandbox agent bridge to use a specific model.
filter option to enable bridge to filter model generate calls.responses_api model arg.eval_set_id to log file (unique id for eval set across invocations for the same log_dir).log_format as the log which is being retried.EvalSetStart and EvalSetEnd hook methods.inspect score now supports streaming via the --stream argument.Agent Bridge: Don't use concurrency() for agent bridge interactions (not required for long-running proxy server or cheap polling requests).
concurrency() for agent bridge interactions (not required for long-running proxy server or cheap polling requests).concurrency parameter to exec() to advise whether the execution should be subject to local process concurrency limits.Bugfix: Work around OpenAI breaking change that renamed "find" web search action to "find_in_page" (bump required version of openai package to v1.104.…
as_solver()).openai v1.104.1 (which is now the minimum required version).ThinkChunk types in mistralai v1.9.10.--reasoning-effort parameter (works w/ gpt-oss models).str_to_float() fails.openai package to v1.104.0).Bugfix: Preserve sample list state (e.g. scroll position, selection) across sample open/close.
Agent Bridge: OpenAI Responses API and Anthropic API are now supported alongside the OpenAI Completions API for both in-process and sandbox-based agen
AgentState changes via inspecting model traffic running over the bridge.messages_df().GenerateConfig values for models override bridged agent config.handoff(): Use content_only() filter by default for handoff output and improve detection of new content from handed off to agents.ContentToolUse ("web_search" or "mcp_call")internal field from ChatMessageBase (no longer used).responses_store model arg for explicitly enabling or disabling the responses API.enum typed fields.thought_signature for thought parts.MCPServerConfig and Stdio and HTTP variants).instance option for multiple services of the same type in a single container.polling_interval option for controlling polling interval from sandbox to scaffold (defaults to 2 seconds, overridden to 0.2 seconds for Docker sandbox).completion).--no-epochs-reducer CLI flag for specifying no reducers.match more lenient when numeric matches container markdown formatting.visible option for concurrency() contexts to control display in status bar.sample_init, sandbox, store, and state events.Scoring: Refactor inspect score to call same underlying code as score().
inspect score to call same underlying code as score().Agent Bridge: New context-manager based agent_bridge() that replaces the deprecated bridge() function.
agent_bridge() that replaces the deprecated bridge() function.sandbox_agent_bridge() to integrate with CLI based agents running inside sandboxes.user_prompt() function for getting the last user message from a list of messages.messages_to_openai() and messages_from_openai() functions for converting to and from OpenAI-style message dicts.response_schema option for providing a JSON schema for model output.--log-dir-allow-dirty option.--continue-on-fail option for eval() and eval_set().copy option to score_async() (defaults to True) to control whether the log is deep copied before scoring.inspect_tool_support.eval-retry by delaying their creation until after task creation.CitationBase so that it no longer leads to an invalid schema.background to OpenAI Responses if specified.tool_choice to Anthropic thinking models.Support for PDF input for OpenAI, Anthropic, and Google via new ContentDocument type.
ContentDocument type.openai package.max_tokens is 8k or higher.betas custom model arg (-M) for opting in to beta features.sample_limits() within scorers.score_to_float() function for converting score columns to float values.if_match_etag parameter for write_eval_log() and etag field to EvalLog for safe concurrent log modification.completion property now does not affect the underlying message content.span() function until required for logging.OpenAI: Enable native web_search() tool for GPT-5.
web_search() tool for GPT-5.sample_shuffle for eval retry.MCP: Support for mcp_server_http() (which replaces the deprecated SSE server mode).
timeout generation config option through to API Client.GOOGLE_VERTEX_BASE_URL.background, safety_identifier and prompt_cache_key custom model args (bump required version of openai package to v1.98).client_timeout to 900s when flex processing is enabled.reasoning_effort option to reasoning dict.mcp_server_http() (which replaces the deprecated SSE server mode).authorization to provide OAuth Bearer token for HTTP based servers.list[str] rather than str without crashing.message_limit to 50 when both message_limit and token_limit are None.with pytest.raises, add test for env vars.with pytest.raises, add test for env vars.MockLLM.ANSWER: None of the above. Allow answers to end with full stop (.).inspect_tool_support to use a Unix socket rather than a tcp port for intra-container RPC.background() task is now scoped to the sample lifetime in the presence of retry_on_error.waiting_time from within coroutines spawned from the main sample coroutine.inspect-tool-support reference container to support executing tool code with non-root accounts.reasoning_effort and reasoning_tokens for OpenRouter provider.bridge() no longer causes a recursion error when running a large number of samples with openai modelsmodel_roles are available within task initialization code.Bugfix: Conform to breaking changes in openai package (1.99.2).
reasoning_effort.openai package (1.99.2).sample_shuffle is None (rather than 0) when not specified on the command line.Analysis functions are out of beta (inspect_ai.analysis.beta is deprecated in favor of inspect_ai.analysis).
inspect_ai.analysis.beta is deprecated in favor of inspect_ai.analysis).store for scorers run on existing log files.Remove support for vertex provider as the google-cloud-aiplatform package has deprecated its support for Vertex generative models. Vertex can still be…
vertex provider as the google-cloud-aiplatform package has deprecated its support for Vertex generative models. Vertex can still be used via the native google and anthropic providers.emulate_tools model arg) to OpenAI API compatible providers.--sample-shuffle eval option to control sample shuffling (takes an optional seed for determinism).Added Fireworks AI model provider.
user and http_client custom model arguments.is_mistral model arg for mistral compatible tool calling.hidden_states model arg to get model activations.--max-connections, --max-retries, and --timeout now provide defaults for all models rather than only the main model being evaluated.max_tool_output.files field.log_viewer() data preparation function.full option to samples_df() for reading full sample metadata.EvalConfig column defs to EvalConfiguration._repr_ for EvalLog (print JSON representation of log header).metadata_as() typesafe metadata accessor to ChatMessageBase.model_info() operation.Added display_name property to Task (e.g. for plotting).
display_name property to Task (e.g. for plotting).task_info() operation for data frame preparation.Analysis: model_info() and frontier() operations for data frame preparation.
model_info() and frontier() operations for data frame preparation.ThinkChunk and AudioChunk in package v1.9.3 (which is now the minimum required version).<think> and <internal> tags from tool messages to prevent leakage in multi-agent scenarios where an inner assistant message can be coerced into a tool message.BaseModel types in tool call schemas.OpenAI: Move model classification functions into ModelAPI class so that subclasses can override them.
ModelAPI class so that subclasses can override them.prepare() function for doing common data preparation tasks and log_viewer() operation for adding log viewer URLs to data frames.uuid field (or event_id in analysis data frames).inspect log convertBatch processing API support for OpenAI and Anthropic models.
HookedTransformer models with Inspect.thought=True on content when replaying ContentReasoning back to the model.uuid field and metadata field to Event.message_id field to ToolEvent for corresponding ChatMessageTool.uuid in read_eval_log_sample().keep_in_messages option to AgentSubmit to preserve calls to submit() in message history.Value type to use covariant types (Mapping and Sequence).display parameter to score() to control display type.scored_samples and unscored_samples fields to indicate how many samples were scored and how many were not. The viewer will display these values if there are unscored samples.--resolve-attachments option to inspect log dump.EvalSample (rather than only the summary) to on_sample_end() hook.inspect view bundle.claude-3-7-sonnet-latest.AgentState output field.run_in_background allowing it to properly function outside the context of a task.None out TaskLogger's SampleBufferDatabase after cleaning it up to avoid crashing on subsequent logging attempts.Bugfix: Conform to breaking changes in mistralai package (1.9.1).
--sample_id filter for datasets.score_headline_stderr field in standard evals column definitions.task_name without package namespace by default.order field in messages_df() and events_df().run_samples option to disable running samples (resulting in a log file with status "started" and no samples).--display=log (improved task info formatting, ability to disable rich logging)mistralai package (1.9.1).Inspect View: Fix issue with tab switching when running in VS Code.
Bugfix: Return inner exception from run_sample.
run_sample.MCP: Conform to breaking changes in latest mcp package (1.10.0).
web_browser() and bash_session() to indicate that you must pass an instance explicitly to get distinct processes.Bugfix: Don't raise error on Anthropic cited_text not being a str.
str.Bugfix: Shield critical shutdown code from cancel scope.
OpenAI: Use prefix matching when detecting compatible models for web_search().
web_search().executed_tools field as model output metadata.str returned from on_continue to the model (formerly this was only done if there were no tool calls).background() function for executing work in the background of the current sample.
sample.limit (as opposed to ones bound to context managers or agents).text_editor tool; don't use native tool definition for Haiku 3.5 or Opus 3.0.ContentReasoning blocks for Gemini (regressed in v0.3.104).seed of 0.Web Search: Added provider for Anthropic's internal web search tool.
metadata field for arbitrary additional metadata.ContentData for model specific content blocks.Citation suite of types and included citations in ContentText (supported for OpenAI and Anthropic models).task_args now includes defaulted args (formerly it only included explicitly passed args).retry_connections now defaults to 1.0 (resulting in no reduction in connections across passes).
OpenAI: Work around OpenAI Responses API issue by filtering out leading consecutive reasoning blocks.- with _ when looking up provider environment variables..devcontainer) configuration.trim_messages() now removes any trailing assistant message after compaction.responses_api=False when pass as an OpenAI model config arg.logs.json summary file during bundling.Eval set: Do not read full eval logs into memory at task completion.
OpenAI: Use responses API for codex models.
Eval set: Default max_tasks to the greater of 4 and the number of models being evaluated.
max_tasks to the greater of 4 and the number of models being evaluated.time_limit() and working_limit() context managers for scoped application of time limits.
docker compose concurrency to 2 * os.cpu_count() by default (override with INSPECT_DOCKER_CLI_CONCURRENCY).on_continue message to the model if the model made no tool calls.Enum types in tool arguments.plain mode (no outline, don't expand tables to console width).Exported view() function for running Inspect View from Python.
view() function for running Inspect View from Python.eval() or eval_set().google-genai to 1.16.1 (which includes support for reasoning summaries and is now compatible with the trio async backend).Google: Disable reasoning when reasoning_tokens is set to 0.
reasoning_tokens is set to 0.React agent: Use of submit() tool is now optional.
submit() tool is now optional.is_agent() typeguard function for checking whether an object is an Agent.temperature, top_p, and top_k).tools or tool_choice in requests when emulating tool calling (avoiding a 400 error).<tool_calls> plural from Llama models (as it sometimes uses this instead of <tool_call>).--sample-id is not found in the target dataset (raise error if there are no matches at all).EvalLog and Exception in ColumnError.inspect view bundle, support linking to individual transcript events or messages.Dataframes: events_df() function, improved message reading, log filtering, don't re-sort passed logs
events_df() function, improved message reading, log filtering, don't re-sort passed logsmcp package.submit().Dataframe functions for reading dataframes from log files.
max_tokens option to control tokens used for generate().working_limit after solvers have completed executing (matching behavior of other sample limits).user parameter on to sandboxes if is not None (eases compatibility with older sandbox providers).type in the error message body is "overloaded_error".request() method in v1.78.0 of openai package (now the minimum required version).mcp package (now the minimum required version).input_text and user_prompt properties now read the last rather than first user message.metadata from TaskState rather than from Sample.thought argument being passed to think() tool.text_editor() tool for Claude Sonnet 3.5.span() function for grouping transcript events.
inspect log convert now always fully re-writes log files even of the same format (so that e.g. sample summaries always exist in the converted logs).answer_only and answer_delimiter to control how submitted answers are reflected in the assistant message content.bash() and python() tools.Scoped Limits for enforcing token and message limits using a context manager.
bash_session() tool to provide richer interface to model and to support interactive sessions (e.g. logging in to a remote server).vllm package (improved concurrency and resource utilization).streaming model argument to control whether streaming API is used (by default, streams when using extended thinking).--sample-id option can now include task prefixes (e.g. --sample-id=popularity:10,security:5)).--no-log-realtime option for disabling realtime event logging (live viewing of logs is disabled when this is specified)._resources directories from package (reduces pressure on path lengths for Windows).OpenAI: In responses API, don't pass back assistant output that wasn't part of the output included in the server response (e.g. output generated from
submit() tool).Support for using tools from Model Context Protocol providers.
responses_store model argument to control whether the store option is enabled (it is enabled by default for reasoning models to support reasoning playback).python) for responses API.google-genai (which is now required).reasoning_tokens option for Gemini 2.5 models.reasoning_effort option and capturing reasoning content.reasoning_effort and reasoning_tokens to reasoning field.ToolSource for dynamic tools inputs (can be used in calls to model.generate() and execute_tools())submit() tool.user parameter for running the human agent cli as a given user.model_graded_qa() and model_graded_fact().match() scorer.sample_metadata_as() method to SampleScore.user parameter to connection() method for getting connection details for a given user.datetime, date, time, and Set)Path.--traceback-locals CLI option to print values of local variables in tracebacks.CUDA_VISIBLE_DEVICES environment variable for vLLM provider<think> tag in conversation view if there is no reasoning content.Inspect View: Collapse user messages after 15 lines by default.
Model Roles for creating aliases to models used in a task (e.g. "grader", "red_team", "blue_team", etc.)
on_continue message, including using a dynamic name for the submit tool.metadata field to bridge input for backward compatibility with solver-based bridge.default argument to get_model() to explicitly specify a fallback model if the specified model isn't found.history argument (rather than TaskState) to better handle agent conversation state.2025-03-01-preview as default API version if none explicitly specified.trim_messages() function for pruning messages to fit within model context windows.registry_create() function for dynamic creation of registry objects (e.g. @task, @solver, etc.).chdir option from @task (tasks can no longer change their working directory during execution).INSPECT_EVAL_LOG_FILE_PATTERN environment variable for setting the eval log file pattern.openai/azure/model-name).__future__ as needed.dict keys that are numeric in store diffs.Tools: Restore formerly required (but now deprecated) type field to ToolCall.
type field to ToolCall.reasoning_tokens in total_tokens (they are already included).Eval: Fix an error when attempting to display realtime metrics for an evaluation.
One column per month.