NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI, #2325 most downloaded on PyPI
The LLM Evaluation Framework
Latest stable
4.3.4
Last release 2 days ago
11 Nov 2026
Ships on a steady schedule
a new release about every 1 weeks
Rarely documented
notes for 8 of the last 60 stable releases
1 version withdrawn
withdrawn after publishing
3 years old
533 releases · first in 2023
Drag across the ruler to compare, or tap a release to open it.
Nothing published for this version
Nothing published for this version
If you're using any of the features below, you'll likely see a 50% reduction in code required, especially around ETL for formatting things in and out
If you're using any of the features below, you'll likely see a 50% reduction in code required, especially around ETL for formatting things in and out of DeepEval's ecosystem. This includes:
The first LLM-arena-as-a-Judge metric, now runs a blinded experiment and swaps positions randomly for a fair verdict on which LLM output is better.
Docs: https://deepeval.com/docs/metrics-arena-g-eval
Simply run your loop -> call your agent X number of times -> get your evaluation results. No more trying to fit non-test-case-friendly formats. Instead DeepEval will find your LLM traces automatically to run evals.
from somewhere import your_async_llm_app # Replace with your async LLM app
from deepeval.dataset import EvaluationDataset, Golden
dataset = EvaluationDataset(goldens=[Golden(input="...")])
for golden in dataset.evals_iterator():
# Create task to invoke your async LLM app
task = asyncio.create_task(your_async_llm_app(golden.input))
dataset.evaluate(task)
Docs: https://deepeval.com/docs/evaluation-component-level-llm-evals
Previously you have to define a list of user intentions, profile items, with a ton of more configs to juggle between. Now you can define a list of goldens with a standardized benchmark of scenarios to have turns generated for.
from deepeval.test_case import Turn
from deepeval.simulator import ConversationSimulator
# Create ConversationalGolden
conversation_golden = ConversationalGolden(
scenario="Andy Byron wants to purchase a VIP ticket to a cold play concert.",
expected_outcome="Successful purchase of a ticket.",
user_description="Andy Byron is the CEO of Astronomer.",
)
# Define chatbot callback
async def chatbot_callback(input):
return Turn(role="assistant", content=f"Chatbot response to: {input}")
# Run Simulation
simulator = ConversationSimulator(model_callback=chatbot_callback)
conversational_test_cases = simulator.simulate(goldens=[conversation_golden])
print(conversational_test_cases)
Docs: https://deepeval.com/docs/conversation-simulator
We also updated our docs with more improvements to come 👀
Your coding agent can read these notes before it upgrades. Set up the MCP server →
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
DeepEval's 3.2.6 release focuses on single-vs multi-turn use cases in datasets!
DeepEval's 3.2.6 release focuses on single-vs multi-turn use cases in datasets!
input → output pairs for one-off prompt testing.DeepEval now automatically detects whether a dataset is single-turn or multi-turn based on structure and routes to the appropriate evaluation logic.
Introduced a new concept: conversational goldens, which contains scenario, (and optionally expected_outcome) but not things like input and expected output as with single-turn use cases..
This release is setting the stage for future multi-turn use cases.
Docs here: https://deepeval.com/docs/evaluation-datasets
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
In DeepEval's latest release, we are introducing ArenaGEval, the first ever metric to compare test cases to choose the best performing one based on yo
In DeepEval's latest release, we are introducing ArenaGEval, the first ever metric to compare test cases to choose the best performing one based on your custom criteria.
It looks something like this:
from deepeval import evaluate
from deepeval.test_case import ArenaTestCase, LLMTestCaseParams
from deepeval.metrics import ArenaGEval
a_test_case = ArenaTestCase(
contestants={
"GPT-4": LLMTestCase(
input="What is the capital of France?",
actual_output="Paris",
),
"Claude-4": LLMTestCase(
input="What is the capital of France?",
actual_output="Paris is the capital of France.",
),
},
)
arena_geval = ArenaGEval(
name="Friendly",
criteria="Choose the winter of the more friendly contestant based on the input and actual output",
evaluation_params=[
LLMTestCaseParams.INPUT,
LLMTestCaseParams.ACTUAL_OUTPUT,
],
)
metric.measure(a_test_case)
print(metric.winner, metric.reason)
Docs here: https://deepeval.com/docs/metrics-arena-g-eval
Nothing published for this version
Nothing published for this version
Nothing published for this version
Previously we had great support for single-turn, text evaluation in the form of LLMTestCases, but now we're adding MLLMTestCase, which accepts images:
Previously we had great support for single-turn, text evaluation in the form of LLMTestCases, but now we're adding MLLMTestCase, which accepts images:
from deepeval.metrics import MultimodalGEval
from deepeval.test_case import MLLMTestCaseParams, MLLMTestCase, MLLMImage
from deepeval import evaluate
m_test_case = MLLMTestCase(
input=["Show me how to fold an airplane"],
actual_output=[
"1. Take the sheet of paper and fold it lengthwise",
MLLMImage(url="./paper_plane_1", local=True),
"2. Unfold the paper. Fold the top left and right corners towards the center.",
MLLMImage(url="./paper_plane_2", local=True)
]
)
text_image_coherence = MultimodalGEval(
name="Text-Image Coherence",
criteria="Determine whether the images and text is coherence in the actual output.",
evaluation_params=[MLLMTestCaseParams.ACTUAL_OUTPUT],
)
evaluate(test_cases=[m_test_case], metrics=[text_image_coherence])
Docs here: https://deepeval.com/docs/multimodal-metrics-g-eval
PS. This also includes platform support
<img width="1728" alt="Screenshot 2025-06-19 at 3 46 12 PM" src="https://github.com/user-attachments/assets/b05166be-d407-4a17-8af0-2e86dbdf72b2" />
Nothing published for this version
Nothing published for this version
Nothing published for this version
One column per quarter.