NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI
PyPI, #4527 most downloaded on PyPI
LLM Evaluations
Latest stable
3.10.0
Last release 12 days ago
10 Nov 2026
Ships fairly regularly
a new release about every 2 weeks
Most releases are documented
notes for 39 of the last 60 stable releases
1 version withdrawn
withdrawn after publishing
3 years old
98 releases · first in 2024
Drag across the ruler to compare, or tap a release to open it.
decision-models: support OpenAI Decisions API in Python classification evaluators
Your coding agent can read these notes before it upgrades. Set up the MCP server →
evals: fix PrecisionRecallFScore docstring examples that call the evaluator directly
deps: exclude compromised litellm 1.107.0
evals: reject non-finite precision recall beta
evals: retry RateLimitError in SyncExecutor instead of failing the whole run
evals: deprecate document relevance evaluators
evals: deprecate document relevance evaluators
The Phoenix evals module is graduating out of experimental! You can now install Phoenix evals as a standalone package with pip install arize-phoenix-evals or you can include the new version of phoenix.evals along with the Phoenix install with pip install -U arize-phoenix[evals]. Swapping to the new evals module includes a few small breaking changes which might require some migration work. Details can be found in MIGRATION.md.
phoenix.experimental.evals is being deprecated and will remain in Phoenix for about a month before being removed.
phoenix.evals into phoenix (#2420) (dd3e7b4)evals: deprecate document relevance evaluators
evals: add session-level pii_detection evaluator
## 3.5.1 (2026-08-25) ### Bug Fixes * upgrade to anthropic SDK v1
evals: add retrieval relevance evaluator
phoenix.evals (#2421) (fbd4961)BedrockModel (#2425) (81a720c)evals: add hallucination evaluator
evals: add toxicity gallery template with input/output-agnostic benchmark
evals: add user friction evaluator
evals: compute macro/weighted F-score per class to match sklearn semantics
agents: Move PXI prompt assembly server-side and cache prompts
Nothing published for this version
80bac51: Move @arizeai/phoenix-otel to OpenTelemetry JS 3 (@opentelemetry/sdk-trace 3.x, @opentelemetry/api 1.10) so spans created by apps on OpenTele
@arizeai/phoenix-otel to OpenTelemetry JS 3 (@opentelemetry/sdk-trace 3.x, @opentelemetry/api 1.10) so spans created by apps on OpenTelemetry JS 3 are recorded after register(). register() now returns the @opentelemetry/sdk-trace TracerProvider (still exported as the NodeTracerProvider type), which no longer has a register() method. Custom spanProcessors must come from @opentelemetry/sdk-trace and take an options object, e.g. new BatchSpanProcessor({ exporter }); BatchSpanProcessor and SimpleSpanProcessor are now re-exported from @arizeai/phoenix-otel. See the package's MIGRATION.md. A tracerProvider passed to @arizeai/phoenix-client evaluateExperiment() must now be an @opentelemetry/sdk-trace 3.x TracerProvider. register() no longer applies the OTEL_TRACES_SAMPLER, OTEL_SPAN_*, OTEL_ATTRIBUTE_*, or OTEL_BSP_* environment variables. Node.js 22.15 or newer is now required.add Python 3.14 support (except Windows)
evals: deprecate evals 1.0 docs
evals: add conciseness classification evaluator
This PR was generated with Release Please. See documentation.
<!-- CURSOR_SUMMARY -->
add tool response handling evaluator template
This PR was generated with Release Please. See documentation.
<!-- CURSOR_SUMMARY -->
add FaithfulnessEvaluator and deprecate HallucinationEvaluator
Enable sync- and async- client kwargs for LLM constructor
0128562: fix(phoenix-evals): keep metadata returned by custom evaluator functions wrapped with createEvaluator
metadata returned by custom evaluator functions wrapped with createEvaluatorb00c105: Classification evaluators, including every built-in, can now run on AI SDK decision models such as openai.decisionModel("gpt-6-luna"). The la
b00c105: Classification evaluators, including every built-in, can now run on AI SDK decision models such as openai.decisionModel("gpt-6-luna"). The label comes from experimental_decide, with label probabilities (and the provider's confidence, when reported) in metadata. The rendered prompt is sent as plain text, and refusals throw the AI SDK's Experimental_DecisionRefusalError. Decide spans now record the decision state, question and answer. New exports: isDecisionModel and the DecisionModel type.
This replaces the evaluation model support from the previous release, and changes what evaluation models such as TypeSafe's Jev are sent. They now go through experimental_decide too. Message prompts are joined into plain text instead of being serialized as JSON state.
ai is pinned to 7.0.137 and @ai-sdk/otel to 1.0.137, because the decision API is experimental and has changed in patch releases, and @ai-sdk/otel depends on one exact ai version. Use @ai-sdk/openai 4.0.89 or later with OpenAI decision models. Earlier versions receive the state as JSON-encoded text instead of plain text.
handle None values for top_p and temperature in AnthropicModel
b7b7dfb: Classification evaluators now accept AI SDK evaluation models such as TypeSafe's Jev. When an evaluation model is passed as model, the classi
model, the classification is routed through experimental_evaluate as a single choice question instead of generateObject. Results carry a label and score but no explanation, since evaluation models do not generate text.This PR was generated with Release Please. See documentation.
<!-- CURSOR_SUMMARY -->
d67ea3f: Deprecate createDocumentRelevanceEvaluator in favor of createRetrievalRelevanceEvaluator. Rename documentText to context and the unrelated la…
createCompletenessEvaluator to judge whether every active user request in a conversation was actually completed.createDocumentRelevanceEvaluator in favor of
createRetrievalRelevanceEvaluator. Rename documentText to context and the
unrelated label to irrelevant. The deprecated factory and its types will be
removed in the next major release.Coroutine Fn support for phoenix.evals.create_evaluator
<img width="1858" height="1045" alt="coroutine_evaluator" src="https://github.com/user-attachments/assets/331d425f-bc03-434b-bc18-34ad1ac9317f" />
llm_classify (#2068) (3233d56)188ce28: feat(phoenix-otel): Upgrade openinference-vercel tracing dependency to support decision model tracing
evals: add document relevance evaluator
This PR was generated with Release Please. See documentation.
<!-- CURSOR_SUMMARY -->
## 2.1.0 (2025-09-24) ### Features * Add Azure provider support (#9605) (edf448a) ### Bug Fixes * Remove openai sdk retries (#9587) (08d78f9) * Use OI
v1/traces HTTP endpoint to handle ExportTraceServiceRequest (3c94dea)v1/traces HTTP endpoint to handle ExportTraceServiceRequest (#1968) (3c94dea)## 2.0.1 (2025-09-17) ### Documentation * fix docs preview import paths
evals: add rate limiting to llm methods
llm_classify and llm_generate interfaces (#1974)llm_generate (#1965) (5999133)llm_classify and llm_generate interfaces (#1974) (9fd35a1)evals: add evaluate_dataframe function
evals: make evaluator input schema more explicit and discoverable
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
One column per month.