NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #2199 most downloaded on PyPI
Framework for large language model evaluations
Latest stable
0.3.289
Last release 10 days ago
10 Nov 2026
Ships on a steady schedule
a new release about every 1 weeks
Nearly every release is documented
notes for 60 of the last 60 stable releases
2 versions withdrawn
withdrawn after publishing
3 years old
266 releases · first in 2024
Drag across the ruler to compare, or tap a release to open it.
Anthropic: Improved merging of beta headers.
`ask_user()` tool: model can solicit a structured answer from the operator.
ask_user() tool: model can solicit a structured answer from the operator.notify_user() tool: model can send status notifications to the operator.request_input() public API: programmatic structured prompts from solvers, agents, or tools, using the same dispatch surfaces as ask_user().<think> blocks without exposing inner reasoning text as visible model output.Your coding agent can read these notes before it upgrades. Set up the MCP server →
OpenAI: Backfill required 'query' field when service only provides 'queries'.
model_event_sink option for bridge to control order of model event emission (used to nest model events properly in agent spans).Agent Intervention which provides the ability observe a running agent, interrupt it, and redirect it with follow-up messages.
@tool(parallel=True). Builtin tools that are atomic per call (e.g. bash, python, memory, read_file, web_search) are now registered with parallel=True.--reasoning-effort now works uniformly across Claude 3.7–4.5 and Gemini 2.5. Inspect automatically bridges effort to a budget_tokens / thinking_budget value using a fixed table (minimal=2048, low=4096, medium=10000, high=16000, xhigh/max=32000). Previously these models silently ignored reasoning_effort.minimal / xhigh / max) to the low / medium / high tier accepted by upstream APIs for Groq, Ollama, and SageMaker.max effort to xhigh before forwarding (OpenRouter does not accept max).max → xhigh mapping for reasoning_effort.should_retry() so the model-layer tenacity loop can retry network failures that escape the SDK's one-shot inline retry.run_coroutine() (and the sync log/analysis helpers built on it) now honour INSPECT_ASYNC_BACKEND=trio when called with no running event loop, rather than always using asyncio.AsyncFilesystem: Add iter_files() and iter_dirs() methods.
iter_files() and iter_dirs() methods.pass_k reducer for computing the probability that all k epoch attempts succeed (τ-bench reliability metric).get_model() with role.redacted=False when reasoning content, summary, and encrypted exists.responses_phase parameter.id properly).fastapi and uvicorn are now required dependencies.Config: Add inspect log export-config command to export a run config from an existing log file.
inspect log export-config command to export a run config from an existing log file.get_file() and exists() methods.Scanners: Declare Scanner import in a way that's compatible with pyright type checking.
OpenAI: Add GPT 5.5 as computer use model and exclude 'chat' and 'instant' models from computer use.
reasoning_details in OpenAI-compatible responses.extra_body fields from Message response.openrouter/anthropic/* models.top_k correctly for Nova models.prompt_logprobs support in chat mode via GenerateConfig, parse prompt logprobs from completion mode responses, enabling perplexity() and target_perplexity() scorers end-to-end.--adaptive-connections is now enabled by default (defaults to 100 per model connection).count_tokens() (they are already retried and gated by max_samples).suspend_token_limit() context manager for suspending token tracking and limit enforcement within a scope.hf_dataset retries transient Hugging Face errors (rate limits, timeouts, Hub-unreachable cache misses) up to 3 times (5 in CI) with exponential backoff. Pass retry=False to disable.str() coercion.None is treated (converted to "").hf_dataset(..., shuffle=True) to EvalDataset.shuffled.ToolError if there is a null byte in command input.match(numeric=True) no longer matches digit-substrings (e.g. target 5 against 25); now correctly handles negative, decimal, and scientific-notation targets, and recognises unicode-formatted numbers (unicode minus, vulgar fractions like ½, Chinese numerals, fullwidth digits) in both targets and model output.match(numeric=True, location="exact") is now strict — values like "5 some text" no longer match target "5".evals_df() column name when there are multiple reducers.registry_add()).--run-config option to inspect eval for single-file run configuration.eval_set (CLI --scanner / ScannerConfig). Scans incrementally as logs land, reuses prior results across resumes, and renders progress alongside the existing eval view.IndexError inside resolve_tasks after passing an empty task list to eval.score_display argument to eval_set() function.log_file_info() robust to non-standard filenames; added log_file_info_async() / log_files_from_ls_async() so view-server header reads don't block the event loop.inspect_ai module.INSPECT_PY_LOGGER_FORMAT env var (rich/plain/json) for non-TTY-friendly single-line console logs.COLUMNS and LINES for dumb terminals.GenerateConfig fields with an error.None as default (e.g.x: dict = None).OSError does not include .strerror and .filename[Content, str]."" as an answer for `basic_agent().{submit} in react() agent prompt templates (rather than using .format).pd.NA when converting scores to float in analysis df functions.summaries.json when re-scoring or converting logs.fail_on_error fractional threshold now uses the sliced sample count multiplied by epochs (matching the end-of-run check)eval_retry preserves SampleScore.scorer attribution on restored samples (was None instead of the scorer name).time_limit() no longer masks exceptions raised after the deadline (e.g. from finally blocks) with LimitExceededError; the original exception now propagates.multiple_choice()/answer()/choice() now accept lowercase letters, prefer the last ANSWER: occurrence (matching the CoT "last line" instruction), parse "A, B and C" lists, and handle multi-answer targets with separators (Target("A,B")).max_score reducer's dict/list paths now NaN-filter per key/index, making results order-independent (NaN previously kept whichever element came first).pass_at(k) now returns the unscored NaN sentinel when fewer than k epochs were scored (previously inflated to 1.0).create_reducers no longer rewrites custom reducer names ending in _<digits> (e.g. "top_5") into the built-in _k shorthand.multi_scorer returns Score.unscored() when every sub-scorer declines to score (previously crashed with IndexError).at_least / pass_at reducer names now correctly include the _k suffix on the returned reducer (previously leaked onto the module-global factory).model_graded_qa() / model_graded_fact() — default grade pattern now extracts the last GRADE: $LETTER in grader output.f1() / exact() — targets with leading whitespace are no longer silently skipped.math() — brace-delimited set answers like {1, 2} are now compared as multisets.math() — model outputs containing \boxed{inf} / Infinity / -inf no longer crash the scorer with OverflowError.grouped() — raise instead of silently overwriting a per-group metric when a group name collides with all_label.stderr(cluster=...) — return 0 with a single cluster (was NaN/inf from divide-by-zero).value_to_float() — reject "nan"/"inf" string values so a single non-finite Score doesn't poison accuracy() and friends.inspect log recover — preserve sample.uuid for crashed in-progress samples (initial buffer summary now carries state.uuid; recovery synthesizes a fallback uuid for legacy buffer rows).Anthropic: Skip the top-level cache_control auto-caching field on Bedrock and Vertex where they are not supported.
cache_control auto-caching field on Bedrock and Vertex where they are not supported.tool_call_id in tool responses (parallel tool calling).reasoning_effort for Grok 4 models.ToolError when timeout error occurs in MCP tool call.IncompleteBody errors during multipart uploads under concurrent flushes.list[ContentText].Inspect View: Fix extraneous console errors
Google: Support Gemini 3+ native web search and code execution alongside function tools.
trust_remote_code model argument (defaults to False).vllm-completions model provider that uses completions rather than chat endpoint.retry_immediate now defaults to True. Pass retry_immediate=False (or --no-retry-immediate) to restore the previous batch-retry behavior.max_tasks now defaults to the greater of 10 and the number of models being evaluated (was previously 4).model_usage and role_usage from previous log when performing retries (matches existing eval-retry behavior).usage.input_tokens (currently OpenAI Responses with store=false + include=["reasoning.encrypted_content"]).ToolEvent.message_id to reference the correct ChatMessageTool when multiple tool calls occur in one assistant turn.Add adaptive connections option to automatically tune model API concurrency between configurable bounds based on rate-limit feedback.
fail_on_error threshold).media_resolver() context manager for scoped URI resolution for media reading (images, audio, etc.).download() and gdrive_download() helpers for fetching external files with SHA256 verification, caching, and transient-error retry. gdrive_download() requires the optional gdown dependency, installed via pip install inspect_ai[gdown].stop_reason='model_length' in react() agent, force-compact before falling through to the overflow filter.working_time path in SampleSummary columns.scorer and scorer_args to ScoreEventOpenRouter: Escape signature attribute in tag round-trip.
docker pull for service images already present in the local Docker daemon.input_tokens for gpt-5.4, gpt-5.4-pro, gpt-5.5, gpt-5.5-pro (was 778000, now 922000 = 1,050,000 context − 128,000 output).Handle split UTF-16 "lone surrogate" in log message condensation.
Scoring: Allow NaN-at-root unscored sentinel for dict- and list-valued scorers.
Score.unscored() for samples a scorer cannot produce a value for.CompactionAuto by default as the compaction strategy.CompactionAuto stateless — native compaction is attempted on every trigger rather than remembering fallback state.responses_phase model arg to opt in to synthesizing missing Responses API assistant message phase labels while continuing to preserve OpenAI-returned phase metadata by default.user parameter on bash_session, text_editor, and exec_remote now correctly applied server-side; CLI binary and server socket are no longer accessible to the sandbox's default user.set_flow_content() integration point.throttle() decorator.Agents: Add deepagent() — a batteries-included agent with subagent delegation, persistent memory, structured planning, and an opinionated system promp
deepagent() — a batteries-included agent with subagent delegation, persistent memory, structured planning, and an opinionated system prompt. Includes built-in research(), plan(), and general() subagent factories, a task multiplexer tool for delegation, and support for both isolated and forked (prompt-cache-preserving) dispatch modes.todo_write() planning tool for structured task tracking.read_file(), list_files(), and grep() read-only sandbox tools for agents that need filesystem access without write capabilities.readonly parameter to memory() tool for read-only access to shared memory.instance parameter to skill() for independent per-subagent skill stores. Skill names are now validated for uniqueness in skill(), install_skills(), and across deepagent() parent/subagent scopes.PRTSCR (OpenAI vocab) to xdotool Print keysym so key combos like ALT+PRTSCR work correctly.Google: Honor the press_enter default on Gemini's type_text_at action so typed text commits with Enter as the spec specifies.
press_enter default on Gemini's type_text_at action so typed text commits with Enter as the spec specifies.use_chat_template=false for base model evaluation (complements existing HF provider support).perplexity() and target_perplexity() scorers with perplexity_per_token and perplexity_per_seq metrics for evaluating model prediction quality via prompt log probabilities.prompt_logprobs to GenerateConfig and the --prompt-logprobs CLI flag for requesting prompt token log probabilities (vLLM).ToolEvent when an approver rejects or terminates a call so the attempt is visible in the transcript.OpenAI: Gracefully catch when response.tools is null and normalize it to []
Anthropic: Warn when sampling parameters (temperature, etc.) are passed to Opus 4.7.
temperature, etc.) are passed to Opus 4.7.inference_component_name model argument for routing requests to specific inference components on multi-model endpoints.PRINTSCREEN (OpenAI vocab) to xdotool Print keysym so key combos like ALT+PRINTSCREEN work correctly.eval_set with retry_immediate=True that could cause ClosedResourceError when a task entered the retry path while other workers were completing concurrently.Capture compaction strategy params in eval log.
Google: Correct counting for cached input tokens.
OSError from path validation (e.g. ENAMETOOLONG) to the model as a tool error instead of crashing the eval.Anthropic: Auto-detect correct context window and max tokens for Opus 4.7.
xhigh value for effort.max value for reasoning_effort.Eval Set: Display explicit --id in task panel headers when provided.
--id in task panel headers when provided.--retry-immediate option to retry failed tasks immediately without waiting for all tasks to complete, reusing completed samples from the failed run.header_only parameter to write_eval_log() for writing only the header to .eval files without rewriting samples.inspect log recover command for recovering crashed eval logs from the sample buffer database. Recovers both completed (unflushed) and in-progress samples. Automatic recovery is integrated into eval_set() and eval_retry().ModelEvent) when retrying samples.cmd to command.INSPECT_SANDBOX_MAX_READ_FILE_SIZE and INSPECT_SANDBOX_MAX_EXEC_OUTPUT_SIZE environment variables for overriding limits.timeout command.on_before_model_generate() hook.None to NaN during deserialization.comma keysym so key combos like CTRL+, work correctly.sudo package to computer tool Docker image.server_error when server returns non-ChatCompletion (which can occur in some cases for OpenRouter).display="summarized" in thinking configuration.fastapi and uvicorn packages are available.eval_results() producing identical aggregate scores
for multiple instances of the same scorer due to incorrect name
resolution using dimension names instead of scorer names.eval_results() mutating the reducers parameter inside
a loop, causing inconsistent reducer assignment across scorer
instances.JSONRecorder returning condensed ModelEvent.input (empty list) when eval() uses log_format="json".to_uri() encoding @ as %40 in local file paths, breaking round-trip through filesystem()/local_path().answer("word") scorer failing to match Unicode symbol characters (e.g. ☆, ○, ◎).Eval Logs: Ensure that condense_events() is called when re-writing eval logs.
condense_events() is called when re-writing eval logs.NOANSWER instead of INCORRECT when the pattern scorer fails to match.Timelines: Improved detection of forked_at for branches from user or system messages.
forked_at for branches from user or system messages.OpenAI: Add cyber_policy to "content_filter" stop reason
cyber_policy to "content_filter" stop reasonBranchEvent and timeline_branch() to delineate timeline branches.TimelineBranch into TimelineSpan via forked_at property.Google: Update to google-genai v1.69.0 to address type changes (async_http_client can now be None for Vertex with Google Auth).
google-genai v1.69.0 to address type changes (async_http_client can now be None for Vertex with Google Auth).read_approval_policies() function for reading approval policies from a config file.metadata field to Approval which is in turn forwarded to ApprovalEvent.parse_tool_info() to improve performance when there are many tools defined.required field to get_model() for ensuring that model roles are specified.model_roles() function to get model roles for the active task.forked_at detection for forking on non-assistant messages.src/inspect_ai/_view/www/ into ts-mono/apps/inspect/ monorepo (pnpm + Vite + Jest). Built assets are copied to src/inspect_ai/_view/dist/ via a Vite plugin. No user-facing changes.Google: Remove deprecated gemini-3-pro-preview from computer use model check and replace with gemini-3.1-pro-preview in tests and docs.
gemini-3-pro-preview from computer use model check and replace with gemini-3.1-pro-preview in tests and docs.completion_mode for CPT/base models, sending completions-style request payloads with logprobs and prompt_logprobs support.read_timeout and connect_timeout model args.do_sample model arg for overriding default sampling behavior.client_timeout to OpenAICompatibleAPI and VLLMAPI.- (e.g. -0.07") by using the = form for the --text argument.embed_viewer=True, and keep listing.json updated as logs are created.run_multiple silently swallowing task finalisation errors and returning success=True with no results.tool_calls and source when combining assistant messages.math.inf so it never blocks.Google: Fix intermittent FAILED_PRECONDITION error when using native code execution by omitting function calling system instruction hint.
exec_remote to prevent OOM from unbounded subprocess output.timeout and timeout_retry for exec_remote() requests.bind()/listen() in sandbox tools server.approval() context manager and approval arguments to execute_tools() and react() agent.ValueError when sandbox provider is not found.OpenAI: Store readable reasoning text in summary when both text and encrypted reasoning content are provided.
summary when both text and encrypted reasoning content are provided.<think> tags that appear in assistant messages.OpenAI: Support for image output for multimodal modals.
reasoning_effort for Nova models.CompactionEdit and CompactionNative strategies.as_tool() agent wrapper.samples and reductions fields lazily in EvalLog returned by eval().edit_eval_log().tags parameter to Task, merged with eval-level tags at eval time.condense_events/expand_events API pair and EventsData TypedDict for deduplicating and restoring repeated model event inputs and call messages.--max-dataset-memory option to limit the size of datasets held in memory during execution. When exceeded, samples are paged to disk.viewer/ subdirectory, fixing permission issues when serving logs.NotADirectoryError when locating sandbox tools binary so S3 download/build fallbacks run on Python < 3.13.staticx incompatibility with setuptools 82+ (removed pkg_resources).OpenAI: Don't serialize unspecified fields in ResponseCustomToolCallParam.
ResponseCustomToolCallParam.Anthropic: Preserve OAuth beta header when per-request betas are set.
agent_result for timeline spans.exec_remote() now auto-injects sandbox tools CLI if needed.on_sample_event() hook.upload_fileobj for automatic multipart uploads.absolute_file_path (matches local fs behavior).inspect view embed command and --embed-viewer eval-set option to embed a log viewer into a log directory.OpenAI: Fix mypy issues with OpenAI SDK 2.26 (now required).
Anthropic: Handle updated Anthropic compaction not supported error message.
frozendict as fallback; update jsonpath-ng dependency.<summary> from <details> tags in tool views.anonymous and region_name parameters to support credential-free access to public S3 buckets..eval file sizes.Anthropic: Use text_editor_20250728 for all Claude 4.x models per Anthropic docs.
text_editor_20250728 for all Claude 4.x models per Anthropic docs.agent_span_id property to tool events for associating them with their associated agent.Improved naming and typeinfo for exec_remote() output stream events.
exec_remote() output stream events.Events: Rename EventNode to EventTreeNode and SpanNode to EventTreeSpan (old type names will still work at runtime with a deprecation warning).
anthropic package is now required).samples_df() (now 50x faster).config_deserialize().exec_remote() method for async execution of long-running commands.type field to CompactionEvent to record compaction type.cost_limit() context manager for scoped application of cost limits.EventNode to EventTreeNode and SpanNode to EventTreeSpan (old type names will still work at runtime with a deprecation warning).thinking_config) under generation_config in the REST schema.EXPIRED or PARTIALLY_SUCCEEDED state.Bugfix: Fix shutdown hang by draining nest_asyncio event loop.
Inspect View: Fix a regression which affected the display of samples within VSCode.
Added stable_message_ids() function for yielding stable ids based on model content (but always unique within a given conversation).
stable_message_ids() function for yielding stable ids based on model content (but always unique within a given conversation).CompactionEvent for model compactions.<think> tags when included in user messages.--reasoning-history CLI argument (don't parse as boolean).human_cli() with no answer now correctly completes task.Web Search: Fallback to Google CSE provider only when Google CSE environment variables are defined (the CSE service has been deprecated by Google).
<think> tags in assistant message loading (now all done directly by model providers).INSPECT_USE_ZSTD environment variable.Early Stopping: Check for early stopping after sample semaphore is acquired rather than before.
json.dumps for message cache keys (incompatible with BaseModel types).Eval Logs: Improve load time by using JSON in duplicate message cache rather than frozendict.
frozendict.trim_message() to use the same behavior).Google: Provide JSON schema directly rather than converting it to Google Schema type.
ContentReasoning as <think> with attributes to prevent bridge clients from doing a more lossy <think> tag conversion.count_tokens() method.count_tokens() method.yyyy-mm-dd hh:mm:ss format (keep local time zone).Anthropic: Only re-order reasoning blocks for Claude 3 (as we use interleaved thinking for Claude 4).
samples_df().Google: Add streaming model arg to opt-in to streaming generation.
streaming model arg to opt-in to streaming generation.image_input (data URI) in field spec for multimodal tasksComposeConfig directly to Docker sandbox provider.supported_fields parameter from parse_compose_yaml() (packages handle their own validation).Sandbox: parse_compose_yaml() for parsing Docker Compose files into typed configuration for sandbox providers.
parse_compose_yaml() for parsing Docker Compose files into typed configuration for sandbox providers.strict=True. This is required by HF Inference Providers and Fireworks (and possibly others).task status and task stop.Agent Bridge: Consolidate bridged tools implementation into the existing sandbox model proxy service (eliminate Python requirement for using bridged t
{} as value for additionalProperties in tool schema.background parameter as this is OpenAI service-specific.minimal and medium reasoning effort levels for Gemini 3 Flash.max_tokens is greater than 16000.combined_from metadata field when combining consecutive user or assistant messages for call to generate.kwargs to JSON readers (built-in reader and jsonlines reader).async_connection()builtins module rather than __builtins__ when parsing tool function types.Compaction: Compacting message histories for long-running agents that exceed the context window.
count_tokens() method for estimating token usage for messages.ModelInfo for retrieving information about models (e.g. organization, context window, reasoning, release date, etc.)reasoning_details (map onto standard reasoning fields for viewer).IO[bytes] via read_eval_log().shell_command tool calls.skill() tool to make agent skills available to models.
Eval Set: Correct log reuse behavior when epochs and limit change.
RuntimeError.Anthropic: Treat reasoning text as a summary (true for all models after Sonnet 3.7).
reasoning_details field to forward native reasoning replay to models.summary in serialization for agent bridge.@agent functions with no return type decoration.retry_refusals option to retry on stop_reason == "content_filter".choices in EvalSampleSummary.inspect view bundle to publish hf/ prefixed targets to Hugging Face Spaces.Log File column of the samples list.type field to enable lists of types.Eval Set: Defer reading eval samples until they are actually needed (prevents memory overload for large logs being retried).
anthropic/azure).-M streaming=true).Early Stopping API for ending tasks early based on previously scored samples.
az://).cleanup() function at the end of the sample (after scoring) rather than after solvers.disable_retry) for waiting time tracking.--verbosity generation option.reasoning_effort option.temperature, top_p, and logprobs for GPT 5.x models with reasoning disabled.-M stream=false to disable).stream option (disabled by default, use -M stream=true to enable).model option is now used only as a fallback if the request model is not for "inspect" or "inspect/*".metadata field to new eval for eval-retry.Agent Bridge: Don't print serialization warnings when going from Pydantic -> JSON (as we use beta types that can cause warnings even though serializat
Update Plan tool for tracking steps and progress across longer horizon tasks.
--effort) for trading off between response thoroughness and token efficiency.web_fetch tool as part of web_search() implementation (matching capability of other providers that have native web search).caller field for server tool uses (required by package version 0.75, which is now the minimum version).-M conversation_api=False).retry_delay).BridgedToolsSpec and bridged_tools parameter to sandbox_agent_bridge() for exposing host-side Inspect tools to sandboxed agents via MCP protocol.EvalLog objects directly to dataframe functions (samples_df(), evals_df(), messages_df(), events_df()).mcp package version 1.23.0.Memory tool: Added memory() tool and bound it to native definitions for providers that support it (currently only Anthropic).
xai_sdk, which is now required).output_limit as a cap enforced with a circular buffer (rather than a limit that results in killing the process and raising).evals_in_eval example for running Inspect evaluations inside other evaluations.datetime's via DTZ lint rule.nest_asyncio, which is fundamentally incompatible with Python 3.14, to nest_asyncio2, which has explicit 3.14 compatibility.read_eval_log() with JSON log files.Anthropic: Enable interleaved-thinking by default for Claude 4 models.
max_tokens handling to prevent exceeding model max tokens when reasoning tokens are specified.RegistryInfo and registry_info() to the public API.prompt_cache_retention is correctly forwarded by agent bridge to responses API.Inspect View: Truncate display of large sample summary fields to improve performance.
One column per month.