⚙️ New Features

DeepEval's 3.2.6 release focuses on single-vs multi-turn use cases in datasets!

🧩 Support for Single-Turn and Multi-Turn Datasets

• Single-turn datasets: Simple input → output pairs for one-off prompt testing.
• Multi-turn datasets: Full conversation flows with alternating user/assistant turns. Perfect for simulating real chat interactions.

DeepEval now automatically detects whether a dataset is single-turn or multi-turn based on structure and routes to the appropriate evaluation logic.

🧪 Conversational Goldens

Introduced a new concept: conversational goldens, which contains scenario, (and optionally expected_outcome) but not things like input and expected output as with single-turn use cases..

✅ Improvements

• Smarter dataset evaluation routing: Whether single-turn or multi-turn, DeepEval figures it out and builds test cases accordingly.
• Improved multi-turn context preservation: Each conversational turn is maintained during evaluation, giving more accurate multi-turn metrics.

This release is setting the stage for future multi-turn use cases.

Docs here: https://deepeval.com/docs/evaluation-datasets

Metric that is alike LLM Arena is Here

In DeepEval's latest release, we are introducing ArenaGEval, the first ever metric to compare test cases to choose the best performing one based on your custom criteria.

It looks something like this:

from deepeval import evaluate
from deepeval.test_case import ArenaTestCase, LLMTestCaseParams
from deepeval.metrics import ArenaGEval

a_test_case = ArenaTestCase(
    contestants={
        "GPT-4": LLMTestCase(
            input="What is the capital of France?",
            actual_output="Paris",
        ),
        "Claude-4": LLMTestCase(
            input="What is the capital of France?",
            actual_output="Paris is the capital of France.",
        ),
    },
)
arena_geval = ArenaGEval(
    name="Friendly",
    criteria="Choose the winter of the more friendly contestant based on the input and actual output",
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
    ],
)


metric.measure(a_test_case)
print(metric.winner, metric.reason)

Docs here: https://deepeval.com/docs/metrics-arena-g-eval

In DeepEval's latest release, we are introducing multimodal G-Eval, plus 7+ multimodal metrics!

Previously we had great support for single-turn, text evaluation in the form of LLMTestCases, but now we're adding MLLMTestCase, which accepts images:

from deepeval.metrics import MultimodalGEval
from deepeval.test_case import MLLMTestCaseParams, MLLMTestCase, MLLMImage
from deepeval import evaluate

m_test_case = MLLMTestCase(
    input=["Show me how to fold an airplane"],
    actual_output=[
        "1. Take the sheet of paper and fold it lengthwise",
        MLLMImage(url="./paper_plane_1", local=True),
        "2. Unfold the paper. Fold the top left and right corners towards the center.",
        MLLMImage(url="./paper_plane_2", local=True)
    ]
)
text_image_coherence = MultimodalGEval(
    name="Text-Image Coherence",
    criteria="Determine whether the images and text is coherence in the actual output.",
    evaluation_params=[MLLMTestCaseParams.ACTUAL_OUTPUT],
)

evaluate(test_cases=[m_test_case], metrics=[text_image_coherence])

Docs here: https://deepeval.com/docs/multimodal-metrics-g-eval

PS. This also includes platform support

<img width="1728" alt="Screenshot 2025-06-19 at 3 46 12 PM" src="https://github.com/user-attachments/assets/b05166be-d407-4a17-8af0-2e86dbdf72b2" />

In DeepEval's latest release, we are introducing a slight change in how a conversation is evaluated.

Previously we assumed a conversation as as a list of LLMTestCases, which might necessarily be the case. Now a conversational test case is made up of a list of Turns instead, which follows OpenAI's standard messages format:

from deepeval.test_case import Turn

turns = [Turn(role="user", content="...")]

Docs here: https://deepeval.com/docs/evaluation-test-cases#conversational-test-case

Added new loading bars for component-level evals, and deepeval view to see results on Confident AI.