NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #2325 most downloaded on PyPI
The LLM Evaluation Framework
Latest stable
4.3.4
Last release 5 days ago
11 Nov 2026
Ships on a steady schedule
a new release about every 1 weeks
Rarely documented
notes for 8 of the last 60 stable releases
1 version withdrawn
withdrawn after publishing
3 years old
533 releases · first in 2023
Drag across the ruler to compare, or tap a release to open it.
Nothing published for this version
Previously we assumed a conversation as as a list of LLMTestCases, which might necessarily be the case. Now a conversational test case is made up of a
Previously we assumed a conversation as as a list of LLMTestCases, which might necessarily be the case. Now a conversational test case is made up of a list of Turns instead, which follows OpenAI's standard messages format:
from deepeval.test_case import Turn
turns = [Turn(role="user", content="...")]
Docs here: https://deepeval.com/docs/evaluation-test-cases#conversational-test-case
Nothing published for this version
Your coding agent can read these notes before it upgrades. Set up the MCP server →
Added new loading bars for component-level evals, and deepeval view to see results on Confident AI.
Added new loading bars for component-level evals, and deepeval view to see results on Confident AI.
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
DeepTeam is built for LLM security — adversarial testing, attack generation, and vulnerability discovery.
We’re excited to introduce DeepEval v3.0, a major milestone that transforms how you evaluate LLM applications — from complex multi-step agents to simple prompt chains. This release brings component-level granularity, production-ready observability, and simulation tools to empower devs building modern AI systems.
You can now apply DeepEval metrics to any step of your LLM workflow — tools, memories, retrievers, generators — and monitor them in both development and production.
observe()Automatically simulate realistic multi-turn conversations to test your chatbots and agents.
Bootstrapping eval datasets just got easier. Now you can exponentially expand your test cases using LLM-generated variants of existing goldens.
All red teaming functionality now lives in its own focused project: DeepTeam. DeepTeam is built for LLM security — adversarial testing, attack generation, and vulnerability discovery.
pip install deepeval --upgrade
DeepEval v3.0 is more than an evaluation framework — it's a foundation for LLM observability. Whether you're debugging agents, simulating conversations, or continuously monitoring production performance, DeepEval now meets you wherever your LLM logic runs.
Ready to explore? 📚 Full docs at deepeval.com →
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
https://www.deepeval.com/docs/metrics-llm-evals#rubric
https://www.deepeval.com/docs/metrics-llm-evals#rubric
Nothing published for this version
Nothing published for this version
Nothing published for this version
In this release we've cleaned up some dependencies to separate out dev packages, as well as more tracing verbose logs for debugging.
In this release we've cleaned up some dependencies to separate out dev packages, as well as more tracing verbose logs for debugging.
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
> ⚠️ This release introduces breaking changes in preparation for DeepEval v3.0.
⚠️ This release introduces breaking changes in preparation for DeepEval v3.0. Please review carefully and adjust your code as needed.
evaluate() function now has "configs"evaluate() function had 13+ arguments to control display, async behaviors, caching, etc. and it was growing out of control. We've now abstracted it into "configs" instead:from deepeval.evaluate.configs import AsyncConfig
from deepeval import evaluate
evaluate(..., async_config=AsyncConfig(max_concurrent=20))
Full docs here: https://www.deepeval.com/docs/evaluation-running-llm-evals#configs-for-evaluate
This shouldn't be a surprised but, DeepTeam now takes care of everything red teaming related, for the foreseeable future. Docs here: https://trydeepteam.com
Nested components are a mess to evaluate. In this version in preparation for v3.0, we introduced dynamic evals, where you can apply a different set of metrics for different components in your LLM application:
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.tracing import observe, update_current_span_test_case
@observe(metrics=[AnswerRelevancyMetric()])
def complete(query: str):
response = openai.ChatCompletion.create(model="gpt-4o", messages=[{"role": "user", "content": query}]).choices[0].message["content"]
update_current_span_test_case(
test_case=LLMTestCase(input=query, output=response)
)
return response
Full docs here: https://www.deepeval.com/docs/evaluation-running-llm-evals#setup-tracing-highly-recommended
Nothing published for this version
Cleaned up dependencies for upcoming 3.0 release:
Cleaned up dependencies for upcoming 3.0 release:
Removed the automatic updates, it is now opt-in: https://www.deepeval.com/docs/miscellaneous
Removed instructor, double checked and it wasn't used anywhere
Removed LlamaIndex and moved it to optional, only needed for one module
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
The latest conversation simulator simulates fake user interactions to generate conversations on your behalf. These conversations can be used for evalu
The latest conversation simulator simulates fake user interactions to generate conversations on your behalf. These conversations can be used for evaluation right afterwards, and is similar to the goldens synthesizer. Docs here: https://docs.confident-ai.com/docs/evaluation-conversation-simulator
Nothing published for this version
Nothing published for this version
Migrated default provider models to support Synthesizer
What's New 🔥
deepeval < 2.5.6 might need to update importsNothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Custom prompt template overriding for all RAG metrics. This was introduced for folks using weaker models for evaluation, or just models in general tha
deepeval's metrics are built around. You can still use your favorite metrics and algorithms, but now with a custom template if required. Example here: https://docs.confident-ai.com/docs/metrics-answer-relevancy#customize-your-templatesave_as() for datasets to save test cases as well: https://docs.confident-ai.com/docs/evaluation-datasets#save-your-datasetSynthesizerDAGMetric: https://docs.confident-ai.com/docs/metrics-dagNothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
One column per quarter.