PyPI · #3960 most downloaded on PyPI

ragas

0.4.31.0M downloads/movibrantlabsai/ragas ↗New releases in your daily digest, changelog included.

Evaluation framework for RAG and LLM applications

Last release 8 months ago
13 Jan 2026
Ships on a steady schedule
a new release about every 2 weeks
Nearly every release is documented
notes for 58 of the last 60 stable releases
1 version withdrawn
withdrawn after publishing
3 years old
88 releases · first in 2023

Release timeline

88 releases since 2023

2023202420252026

20232026

One column per quarter.

Releases

0.1.07 Feb 2024
add initial embeddings.encode_kwargs by @lucasiscovici in https://github.com/explodinggradients/ragas/pull/371
What's Changed

add initial embeddings.encode_kwargs by @lucasiscovici in https://github.com/explodinggradients/ragas/pull/371

feat(testset_generator): support error catching in generation process by @yuukidach in https://github.com/explodinggradients/ragas/pull/368

added class for FastEmbed by @tinomaxthayil in https://github.com/explodinggradients/ragas/pull/379

Updated NLI Verdicts to Binary Format by @lucasiscovici in https://github.com/explodinggradients/ragas/pull/384

added docs for ragas embeddings by @tinomaxthayil in https://github.com/explodinggradients/ragas/pull/386

fix context in _answer_correctness.py by @lucasiscovici in https://github.com/explodinggradients/ragas/pull/385

docs: AI copilot implementation by @nickscamara in https://github.com/explodinggradients/ragas/pull/294

fix: faithfulness for evasive/reject answers by @shahules786 in https://github.com/explodinggradients/ragas/pull/392

Added prompt class by @tinomaxthayil in https://github.com/explodinggradients/ragas/pull/388

Context precision with ground truth by @shahules786 in https://github.com/explodinggradients/ragas/pull/391

Added note to answer_relevance by @shahules786 in https://github.com/explodinggradients/ragas/pull/399

Import evaluate function before patching RagasLLM, fixes #301 by @HySoaKa in https://github.com/explodinggradients/ragas/pull/377

Fix: Added ground_truths arg type-hint info in the eval function by @MANISH007700 in https://github.com/explodinggradients/ragas/pull/404

Typo and phrasing correction by @TruscaPetre in https://github.com/explodinggradients/ragas/pull/408

Small improvements to documentation custom LLMs by @TruscaPetre in https://github.com/explodinggradients/ragas/pull/410

feat(metrics): skip embedding process when the corresponding weight is zero by @yuukidach in https://github.com/explodinggradients/ragas/pull/409

added prompt objects to metrics by @tinomaxthayil in https://github.com/explodinggradients/ragas/pull/401

feat: Automatic Prompt adaptation [language] by @shahules786 in https://github.com/explodinggradients/ragas/pull/407

Update _faithfulness.py by @Robs1999 in https://github.com/explodinggradients/ragas/pull/416

fix(metrics): avoid dividing by zero in correctness by @yuukidach in https://github.com/explodinggradients/ragas/pull/421

Feature/doc update ragas score by @prufrock349 in https://github.com/explodinggradients/ragas/pull/417

Docs: Prompts and prompt adaptation by @shahules786 in https://github.com/explodinggradients/ragas/pull/419

fix(metrics): wrong usage of prompt by @yuukidach in https://github.com/explodinggradients/ragas/pull/422

fix: context_utilization name override by @jjmachan in https://github.com/explodinggradients/ragas/pull/424

feat/executor: make ragas faster and more robust to run by @jjmachan in https://github.com/explodinggradients/ragas/pull/390

fix: Import PrompValue by @shahules786 in https://github.com/explodinggradients/ragas/pull/427

fix: add uuid to analytics by @shahules786 in https://github.com/explodinggradients/ragas/pull/428

feat: added async version for json_load by @jjmachan in https://github.com/explodinggradients/ragas/pull/430

fix: comment error in analytics by @jjmachan in https://github.com/explodinggradients/ragas/pull/433

fix: payload in analytics by @shahules786 in https://github.com/explodinggradients/ragas/pull/434

fix: tests for analytics by @jjmachan in https://github.com/explodinggradients/ragas/pull/435

Docs: tracing and logging by @shahules786 in https://github.com/explodinggradients/ragas/pull/436

feat: global prompt adaptation by @shahules786 in https://github.com/explodinggradients/ragas/pull/432

chore: remove mac artifacts from store by @jjmachan in https://github.com/explodinggradients/ragas/pull/438

fix: vary temperature based on strictness by @shahules786 in https://github.com/explodinggradients/ragas/pull/437

docs: custom llms by @shahules786 in https://github.com/explodinggradients/ragas/pull/442

feat: improving testset generation by @jjmachan in https://github.com/explodinggradients/ragas/pull/381

Fix: context_recall AttributeError by @0ENZO in https://github.com/explodinggradients/ragas/pull/453

Update faithfulness.md by @francesco-defalco in https://github.com/explodinggradients/ragas/pull/451

Fixing typo in faithfulness by @jamesbraza in https://github.com/explodinggradients/ragas/pull/450

fix: context_recall example by @shahules786 in https://github.com/explodinggradients/ragas/pull/458

fix: answer relevancy parsing by @shahules786 in https://github.com/explodinggradients/ragas/pull/461

feat: MVP for the new TestsetGenerator - SimpleEvolution by @jjmachan in https://github.com/explodinggradients/ragas/pull/464

fix: clean up embeddings for ragas and add docs for azure embeddings by @jjmachan in https://github.com/explodinggradients/ragas/pull/477

fix: polishing raise_exception and max_workers by @jjmachan in https://github.com/explodinggradients/ragas/pull/486

Convert prompts in testset generation by @shahules786 in https://github.com/explodinggradients/ragas/pull/484

feat(testset generation): implement new TestsetGenerator by @jjmachan in https://github.com/explodinggradients/ragas/pull/487

doc: add community page by @shahules786 in https://github.com/explodinggradients/ragas/pull/488

Fix ValueError: diag requires an array of at least two dimensions #474 by @herman159 in https://github.com/explodinggradients/ragas/pull/475

doc: modify prompt adaptation by @shahules786 in https://github.com/explodinggradients/ragas/pull/490

docs: theming community by @jjmachan in https://github.com/explodinggradients/ragas/pull/491

fix: CP example by @shahules786 in https://github.com/explodinggradients/ragas/pull/489

feat(testset): ported simple, multi_context and reasoning evolutions by @jjmachan in https://github.com/explodinggradients/ragas/pull/496

fix: cleanup old testset generator by @jjmachan in https://github.com/explodinggradients/ragas/pull/500

feat: added tracking of minimal stuff in testset generation by @jjmachan in https://github.com/explodinggradients/ragas/pull/509

feat: added generate_from_langchain function by @jjmachan in https://github.com/explodinggradients/ragas/pull/511

docs: updated testset generation by @shahules786 in https://github.com/explodinggradients/ragas/pull/510

fix: answer_correctness embedding by @jjmachan in https://github.com/explodinggradients/ragas/pull/513

added docs for customisation by @tinomaxthayil in https://github.com/explodinggradients/ragas/pull/514

fix: added rate_limter to langchain wrapper by @jjmachan in https://github.com/explodinggradients/ragas/pull/517

Update testset_generation.md by @Uzarel in https://github.com/explodinggradients/ragas/pull/516

fix: reshape matrix by @shahules786 in https://github.com/explodinggradients/ragas/pull/519

fix (metrics): changed ground_truths -> ground_truth by @jjmachan in https://github.com/explodinggradients/ragas/pull/521

docs: fix spelling by @shahules786 in https://github.com/explodinggradients/ragas/pull/522

fix: removed old prompt by @shahules786 in https://github.com/explodinggradients/ragas/pull/523

feat(testset): added conditional evolution by @jjmachan in https://github.com/explodinggradients/ragas/pull/524

fix: derive answer from relevant context by @shahules786 in https://github.com/explodinggradients/ragas/pull/525

feat(executor): remove the need to have both sync and async versions of the function by @jjmachan in https://github.com/explodinggradients/ragas/pull/527

feat: seed questions using keyphrases by @shahules786 in https://github.com/explodinggradients/ragas/pull/526

docs: updated contributors by @jjmachan in https://github.com/explodinggradients/ragas/pull/532

feat: configure retries and timeouts with evaluations and testset generators by @jjmachan in https://github.com/explodinggradients/ragas/pull/534

Fix case for few shots of context precision metric by @HuskyDanny in https://github.com/explodinggradients/ragas/pull/541

Updated imports, added dotenv by @aevedis in https://github.com/explodinggradients/ragas/pull/539

feat(executor): run executor in different thread to keep things clean by @jjmachan in https://github.com/explodinggradients/ragas/pull/544

add normalization factor to answer similarity score calculation by @enestydarealmc in https://github.com/explodinggradients/ragas/pull/538

Feat: prompt adaptation for test data generation by @shahules786 in https://github.com/explodinggradients/ragas/pull/530

fix:handle non dict output by @shahules786 in https://github.com/explodinggradients/ragas/pull/547

docs: adaptation for test set generator by @shahules786 in https://github.com/explodinggradients/ragas/pull/548

Fix: ensure quality of adapted prompts by @shahules786 in https://github.com/explodinggradients/ragas/pull/546

fix: raise warning for invalid JSON by @shahules786 in https://github.com/explodinggradients/ragas/pull/550

docs: adding code highlighting by @jjmachan in https://github.com/explodinggradients/ragas/pull/552

fix: wrap langchain embeddings automatically by @jjmachan in https://github.com/explodinggradients/ragas/pull/553

Added documentation page: Using Ragas with Athina for production performance metrics by @shivsak in https://github.com/explodinggradients/ragas/pull/558

fix: proper error message and fixed hf embedding error by @jjmachan in https://github.com/explodinggradients/ragas/pull/561

fix: answer_correctness doesn't reset stuff properly by @jjmachan in https://github.com/explodinggradients/ragas/pull/562

docs: fix issue template by @jjmachan in https://github.com/explodinggradients/ragas/pull/565

docs: fix issue template by @jjmachan in https://github.com/explodinggradients/ragas/pull/566

fix: handle non dict cases by @shahules786 in https://github.com/explodinggradients/ragas/pull/564

fix wrong type annotations and nonexistent types by @stydxm in https://github.com/explodinggradients/ragas/pull/560

fix: raise ExceptionInRunner if executor faces any issues by @jjmachan in https://github.com/explodinggradients/ragas/pull/569

New Contributors

@lucasiscovici made their first contribution in https://github.com/explodinggradients/ragas/pull/371

@nickscamara made their first contribution in https://github.com/explodinggradients/ragas/pull/294

@HySoaKa made their first contribution in https://github.com/explodinggradients/ragas/pull/377

@TruscaPetre made their first contribution in https://github.com/explodinggradients/ragas/pull/408

@Robs1999 made their first contribution in https://github.com/explodinggradients/ragas/pull/416

@prufrock349 made their first contribution in https://github.com/explodinggradients/ragas/pull/417

@0ENZO made their first contribution in https://github.com/explodinggradients/ragas/pull/453

@francesco-defalco made their first contribution in https://github.com/explodinggradients/ragas/pull/451

@jamesbraza made their first contribution in https://github.com/explodinggradients/ragas/pull/450

@herman159 made their first contribution in https://github.com/explodinggradients/ragas/pull/475

@Uzarel made their first contribution in https://github.com/explodinggradients/ragas/pull/516

@HuskyDanny made their first contribution in https://github.com/explodinggradients/ragas/pull/541

@aevedis made their first contribution in https://github.com/explodinggradients/ragas/pull/539

@enestydarealmc made their first contribution in https://github.com/explodinggradients/ragas/pull/538

@shivsak made their first contribution in https://github.com/explodinggradients/ragas/pull/558

@stydxm made their first contribution in https://github.com/explodinggradients/ragas/pull/560

Full Changelog: https://github.com/explodinggradients/ragas/compare/v0.0.22...v0.1.0

0.1.0rc125 Jan 2024pre-release

add initial embeddings.encode_kwargs by @lucasiscovici in https://github.com/explodinggradients/ragas/pull/371

What's Changed

add initial embeddings.encode_kwargs by @lucasiscovici in https://github.com/explodinggradients/ragas/pull/371
feat(testset_generator): support error catching in generation process by @yuukidach in https://github.com/explodinggradients/ragas/pull/368
added class for FastEmbed by @tinomaxthayil in https://github.com/explodinggradients/ragas/pull/379
Updated NLI Verdicts to Binary Format by @lucasiscovici in https://github.com/explodinggradients/ragas/pull/384
added docs for ragas embeddings by @tinomaxthayil in https://github.com/explodinggradients/ragas/pull/386
fix context in _answer_correctness.py by @lucasiscovici in https://github.com/explodinggradients/ragas/pull/385
docs: AI copilot implementation by @nickscamara in https://github.com/explodinggradients/ragas/pull/294
fix: faithfulness for evasive/reject answers by @shahules786 in https://github.com/explodinggradients/ragas/pull/392
Added prompt class by @tinomaxthayil in https://github.com/explodinggradients/ragas/pull/388
Context precision with ground truth by @shahules786 in https://github.com/explodinggradients/ragas/pull/391
Added note to answer_relevance by @shahules786 in https://github.com/explodinggradients/ragas/pull/399
Import evaluate function before patching RagasLLM, fixes #301 by @HySoaKa in https://github.com/explodinggradients/ragas/pull/377
Fix: Added ground_truths arg type-hint info in the eval function by @MANISH007700 in https://github.com/explodinggradients/ragas/pull/404
Typo and phrasing correction by @TruscaPetre in https://github.com/explodinggradients/ragas/pull/408
Small improvements to documentation custom LLMs by @TruscaPetre in https://github.com/explodinggradients/ragas/pull/410
feat(metrics): skip embedding process when the corresponding weight is zero by @yuukidach in https://github.com/explodinggradients/ragas/pull/409
added prompt objects to metrics by @tinomaxthayil in https://github.com/explodinggradients/ragas/pull/401
feat: Automatic Prompt adaptation [language] by @shahules786 in https://github.com/explodinggradients/ragas/pull/407
Update _faithfulness.py by @Robs1999 in https://github.com/explodinggradients/ragas/pull/416
fix(metrics): avoid dividing by zero in correctness by @yuukidach in https://github.com/explodinggradients/ragas/pull/421
Feature/doc update ragas score by @prufrock349 in https://github.com/explodinggradients/ragas/pull/417
Docs: Prompts and prompt adaptation by @shahules786 in https://github.com/explodinggradients/ragas/pull/419
fix(metrics): wrong usage of prompt by @yuukidach in https://github.com/explodinggradients/ragas/pull/422
fix: context_utilization name override by @jjmachan in https://github.com/explodinggradients/ragas/pull/424
feat/executor: make ragas faster and more robust to run by @jjmachan in https://github.com/explodinggradients/ragas/pull/390
fix: Import PrompValue by @shahules786 in https://github.com/explodinggradients/ragas/pull/427
fix: add uuid to analytics by @shahules786 in https://github.com/explodinggradients/ragas/pull/428
feat: added async version for json_load by @jjmachan in https://github.com/explodinggradients/ragas/pull/430
fix: comment error in analytics by @jjmachan in https://github.com/explodinggradients/ragas/pull/433
fix: payload in analytics by @shahules786 in https://github.com/explodinggradients/ragas/pull/434
fix: tests for analytics by @jjmachan in https://github.com/explodinggradients/ragas/pull/435
Docs: tracing and logging by @shahules786 in https://github.com/explodinggradients/ragas/pull/436
feat: global prompt adaptation by @shahules786 in https://github.com/explodinggradients/ragas/pull/432
chore: remove mac artifacts from store by @jjmachan in https://github.com/explodinggradients/ragas/pull/438
fix: vary temperature based on strictness by @shahules786 in https://github.com/explodinggradients/ragas/pull/437
docs: custom llms by @shahules786 in https://github.com/explodinggradients/ragas/pull/442
feat: improving testset generation by @jjmachan in https://github.com/explodinggradients/ragas/pull/381
Fix: context_recall AttributeError by @0ENZO in https://github.com/explodinggradients/ragas/pull/453
Update faithfulness.md by @francesco-defalco in https://github.com/explodinggradients/ragas/pull/451
Fixing typo in faithfulness by @jamesbraza in https://github.com/explodinggradients/ragas/pull/450
fix: context_recall example by @shahules786 in https://github.com/explodinggradients/ragas/pull/458
fix: answer relevancy parsing by @shahules786 in https://github.com/explodinggradients/ragas/pull/461
feat: MVP for the new TestsetGenerator - SimpleEvolution by @jjmachan in https://github.com/explodinggradients/ragas/pull/464
fix: clean up embeddings for ragas and add docs for azure embeddings by @jjmachan in https://github.com/explodinggradients/ragas/pull/477
fix: polishing raise_exception and max_workers by @jjmachan in https://github.com/explodinggradients/ragas/pull/486
Convert prompts in testset generation by @shahules786 in https://github.com/explodinggradients/ragas/pull/484
feat(testset generation): implement new TestsetGenerator by @jjmachan in https://github.com/explodinggradients/ragas/pull/487
doc: add community page by @shahules786 in https://github.com/explodinggradients/ragas/pull/488
Fix ValueError: diag requires an array of at least two dimensions #474 by @herman159 in https://github.com/explodinggradients/ragas/pull/475
doc: modify prompt adaptation by @shahules786 in https://github.com/explodinggradients/ragas/pull/490
docs: theming community by @jjmachan in https://github.com/explodinggradients/ragas/pull/491
fix: CP example by @shahules786 in https://github.com/explodinggradients/ragas/pull/489
feat(testset): ported simple, multi_context and reasoning evolutions by @jjmachan in https://github.com/explodinggradients/ragas/pull/496
fix: cleanup old testset generator by @jjmachan in https://github.com/explodinggradients/ragas/pull/500
feat: added tracking of minimal stuff in testset generation by @jjmachan in https://github.com/explodinggradients/ragas/pull/509
feat: added generate_from_langchain function by @jjmachan in https://github.com/explodinggradients/ragas/pull/511
docs: updated testset generation by @shahules786 in https://github.com/explodinggradients/ragas/pull/510
fix: answer_correctness embedding by @jjmachan in https://github.com/explodinggradients/ragas/pull/513

New Contributors

@lucasiscovici made their first contribution in https://github.com/explodinggradients/ragas/pull/371
@nickscamara made their first contribution in https://github.com/explodinggradients/ragas/pull/294
@HySoaKa made their first contribution in https://github.com/explodinggradients/ragas/pull/377
@TruscaPetre made their first contribution in https://github.com/explodinggradients/ragas/pull/408
@Robs1999 made their first contribution in https://github.com/explodinggradients/ragas/pull/416
@prufrock349 made their first contribution in https://github.com/explodinggradients/ragas/pull/417
@0ENZO made their first contribution in https://github.com/explodinggradients/ragas/pull/453
@francesco-defalco made their first contribution in https://github.com/explodinggradients/ragas/pull/451
@jamesbraza made their first contribution in https://github.com/explodinggradients/ragas/pull/450
@herman159 made their first contribution in https://github.com/explodinggradients/ragas/pull/475

Full Changelog: https://github.com/explodinggradients/ragas/compare/v0.0.22...v0.1.0rc1

Open source →

0.0.2213 Dec 2023

docs: comparing llms using ragas evaluations by @tinomaxthayil in https://github.com/explodinggradients/ragas/pull/314

What's Changed

docs: comparing llms using ragas evaluations by @tinomaxthayil in https://github.com/explodinggradients/ragas/pull/314
convert context precision to json format by @shahules786 in https://github.com/explodinggradients/ragas/pull/322
fix: faithfulness.md by @kaiyikang in https://github.com/explodinggradients/ragas/pull/328
fix: structure faithfulness output by @shahules786 in https://github.com/explodinggradients/ragas/pull/333
fix: Column remap fix for Dataset evaluation by @tinomaxthayil in https://github.com/explodinggradients/ragas/pull/331
Fix OpenAIEmbeddings reference error in answer_relevancy by @ajinkyavbhandare in https://github.com/explodinggradients/ragas/pull/329
Fix: minor improvements across metrics by @shahules786 in https://github.com/explodinggradients/ragas/pull/337
fix wrong number by @Yongtae723 in https://github.com/explodinggradients/ragas/pull/338
fix: faithfulness for unrelated answering by @shahules786 in https://github.com/explodinggradients/ragas/pull/340
fix: load response as json by @shahules786 in https://github.com/explodinggradients/ragas/pull/341
docs: remove answer_relevancy from llm-docs by @tinomaxthayil in https://github.com/explodinggradients/ragas/pull/343
fix: bedrock llm error when evaluating rag qa validate_api_key by @arm-diaz in https://github.com/explodinggradients/ragas/pull/350
Added integration with Amazon API Gateway + bug fix in _answer_relevance.py by @brunopistone in https://github.com/explodinggradients/ragas/pull/327
feat: improve quality of answer correctness by @shahules786 in https://github.com/explodinggradients/ragas/pull/339
docs: add zeno visualization integration by @cabreraalex in https://github.com/explodinggradients/ragas/pull/359
fix(metrics): avoid getting none in context_recall by @yuukidach in https://github.com/explodinggradients/ragas/pull/355
fix: improved answer relevancy by @shahules786 in https://github.com/explodinggradients/ragas/pull/346
fix(types): fix pyright type issues with latest pyright version by @jjmachan in https://github.com/explodinggradients/ragas/pull/366
Retry for failed json loading by @shahules786 in https://github.com/explodinggradients/ragas/pull/364
fix: handle edge cases in prompt processing by @shahules786 in https://github.com/explodinggradients/ragas/pull/374

New Contributors

@kaiyikang made their first contribution in https://github.com/explodinggradients/ragas/pull/328
@ajinkyavbhandare made their first contribution in https://github.com/explodinggradients/ragas/pull/329
@brunopistone made their first contribution in https://github.com/explodinggradients/ragas/pull/327
@cabreraalex made their first contribution in https://github.com/explodinggradients/ragas/pull/359
@yuukidach made their first contribution in https://github.com/explodinggradients/ragas/pull/355

Full Changelog: https://github.com/explodinggradients/ragas/compare/v0.0.21...v0.0.22

Open source →

0.0.2121 Nov 2023

docs: Import AnswerSimilarity by @nelsonauner in https://github.com/explodinggradients/ragas/pull/296

What's Changed

docs: Import AnswerSimilarity by @nelsonauner in https://github.com/explodinggradients/ragas/pull/296
FIx: testset_generator.py by @ferdinandl007 in https://github.com/explodinggradients/ragas/pull/291
Case-insensitive and whitespace handling for context_precision. by @Pauldevillers in https://github.com/explodinggradients/ragas/pull/297
Fix/tiny refactor by @Yongtae723 in https://github.com/explodinggradients/ragas/pull/305
fix: broken link to metrics by @younes-io in https://github.com/explodinggradients/ragas/pull/311
Convert context_recall to json format by @shahules786 in https://github.com/explodinggradients/ragas/pull/312
fix: add retry logic for OpenAI and Azure OpenAI by @jjmachan in https://github.com/explodinggradients/ragas/pull/315
fix: openai env var load after init and before score also by @jjmachan in https://github.com/explodinggradients/ragas/pull/316
import llama_index gracefully, solves #318 by @ceferisbarov in https://github.com/explodinggradients/ragas/pull/319
Make testgenerator output compatible with evaluate by @tinomaxthayil in https://github.com/explodinggradients/ragas/pull/302

New Contributors

@nelsonauner made their first contribution in https://github.com/explodinggradients/ragas/pull/296
@ferdinandl007 made their first contribution in https://github.com/explodinggradients/ragas/pull/291
@Pauldevillers made their first contribution in https://github.com/explodinggradients/ragas/pull/297
@younes-io made their first contribution in https://github.com/explodinggradients/ragas/pull/311
@ceferisbarov made their first contribution in https://github.com/explodinggradients/ragas/pull/319

Full Changelog: https://github.com/explodinggradients/ragas/compare/v0.0.20...v0.0.21

Open source →

0.0.2015 Nov 2023

docs: update how LLMs are customized in Ragas metrics by @jjmachan in https://github.com/explodinggradients/ragas/pull/219

What's Changed

docs: update how LLMs are customized in Ragas metrics by @jjmachan in https://github.com/explodinggradients/ragas/pull/219
fix: Add RagasLLM for customized test set generation by @chankeith96 in https://github.com/explodinggradients/ragas/pull/242
docs: add import statement in testset generation by @jjmachan in https://github.com/explodinggradients/ragas/pull/249
expose metrics prompts by renaming metric modules by @grauvictor in https://github.com/explodinggradients/ragas/pull/243
fix: added support for google vertexAI by @jjmachan in https://github.com/explodinggradients/ragas/pull/252
Renamed module names and made changes in init to reflect the change by @tinomaxthayil in https://github.com/explodinggradients/ragas/pull/244
added hint in synthetcic data generation by @tinomaxthayil in https://github.com/explodinggradients/ragas/pull/259
feat: RagasEmbeddings by @shahules786 in https://github.com/explodinggradients/ragas/pull/232
Fix: type hints by @shahules786 in https://github.com/explodinggradients/ragas/pull/263
Update issue templates by @shahules786 in https://github.com/explodinggradients/ragas/pull/265
Update evaluation.md typo by @whitewum in https://github.com/explodinggradients/ragas/pull/275
removed ragas score from result object by @tinomaxthayil in https://github.com/explodinggradients/ragas/pull/276
Custom embeddings from HuggingFace (sentence transformer) do not pass the assertion in src/ragas/metrics/_answer_similarity.py by @alebondarenko in https://github.com/explodinggradients/ragas/pull/277
fix: set temperate to near zero value by @shahules786 in https://github.com/explodinggradients/ragas/pull/267
feat: add native support for OpenAI and AzureOpenAI by @jjmachan in https://github.com/explodinggradients/ragas/pull/261
feat: make ragas pip more lighter by @jjmachan in https://github.com/explodinggradients/ragas/pull/283
docs: amazon bedrock (#241) by @arm-diaz in https://github.com/explodinggradients/ragas/pull/246
docs: added vertext ai support by @jjmachan in https://github.com/explodinggradients/ragas/pull/284
chore: pin OpenAI to >1 by @jjmachan in https://github.com/explodinggradients/ragas/pull/285

New Contributors

@chankeith96 made their first contribution in https://github.com/explodinggradients/ragas/pull/242
@grauvictor made their first contribution in https://github.com/explodinggradients/ragas/pull/243
@tinomaxthayil made their first contribution in https://github.com/explodinggradients/ragas/pull/244
@whitewum made their first contribution in https://github.com/explodinggradients/ragas/pull/275
@alebondarenko made their first contribution in https://github.com/explodinggradients/ragas/pull/277

Full Changelog: https://github.com/explodinggradients/ragas/compare/v0.0.19...v0.0.20

Open source →

0.0.1931 Oct 2023

docs: fix typo by @shahules786 in https://github.com/explodinggradients/ragas/pull/214

What's Changed

docs: fix typo by @shahules786 in https://github.com/explodinggradients/ragas/pull/214
feat: context average precision by @shahules786 in https://github.com/explodinggradients/ragas/pull/204
fix: avoid division by zero by @shahules786 in https://github.com/explodinggradients/ragas/pull/218
fix: typecast in TestsetGeneration fails for python3.8 by @jjmachan in https://github.com/explodinggradients/ragas/pull/215
fix: make context_relevancy available by @jjmachan in https://github.com/explodinggradients/ragas/pull/229
feat: add support for llms on AWS Bedrock (#226) by @arm-diaz in https://github.com/explodinggradients/ragas/pull/227
tests: for metrics by @jjmachan in https://github.com/explodinggradients/ragas/pull/233
docs: add context_precision by @shahules786 in https://github.com/explodinggradients/ragas/pull/236
docs: add langfuse notebook by @jjmachan in https://github.com/explodinggradients/ragas/pull/216
Fix typos in langfuse.ipynb by @marcklingen in https://github.com/explodinggradients/ragas/pull/237

New Contributors

@arm-diaz made their first contribution in https://github.com/explodinggradients/ragas/pull/227
@marcklingen made their first contribution in https://github.com/explodinggradients/ragas/pull/237

Full Changelog: https://github.com/explodinggradients/ragas/compare/v0.0.18...v0.0.19

Open source →

0.0.1824 Oct 2023

fix: use llm from baseclass for metrics by @jjmachan in https://github.com/explodinggradients/ragas/pull/189

What's Changed

fix: use llm from baseclass for metrics by @jjmachan in https://github.com/explodinggradients/ragas/pull/189
bugfix: context_precision by @shahules786 in https://github.com/explodinggradients/ragas/pull/194
Fix typos in Eval doc by @mmaysami in https://github.com/explodinggradients/ragas/pull/198
testset generation : Improvements by @shahules786 in https://github.com/explodinggradients/ragas/pull/195
feat: langchain documents support for TestsetGenerator by @jjmachan in https://github.com/explodinggradients/ragas/pull/201
docs: document loaders support by @shahules786 in https://github.com/explodinggradients/ragas/pull/202
feat: support for multiple n in llms by @jjmachan in https://github.com/explodinggradients/ragas/pull/197
feat: llamaIndex llm support by @jjmachan in https://github.com/explodinggradients/ragas/pull/205
Refactor testset_generator.py for newbies to understand easily by @Yongtae723 in https://github.com/explodinggradients/ragas/pull/207
fix: remove limit on pydantic 2.0 by @jjmachan in https://github.com/explodinggradients/ragas/pull/209

New Contributors

@mmaysami made their first contribution in https://github.com/explodinggradients/ragas/pull/198

Full Changelog: https://github.com/explodinggradients/ragas/compare/v0.0.17...v0.0.18

Open source →

0.0.1716 Oct 2023

Testset generation: add gaurdrails to chunk filtering by @shahules786 in https://github.com/explodinggradients/ragas/pull/166

What's Changed

Testset generation: add gaurdrails to chunk filtering by @shahules786 in https://github.com/explodinggradients/ragas/pull/166
Docs testset generation by @shahules786 in https://github.com/explodinggradients/ragas/pull/165
don't warn when context_precision is correctly called by @monstertruck in https://github.com/explodinggradients/ragas/pull/168
fixed chunk validation by @shahules786 in https://github.com/explodinggradients/ragas/pull/169
doc: init docs.ragas.io by @jjmachan in https://github.com/explodinggradients/ragas/pull/170
docs: Add references to ragas by @jjmachan in https://github.com/explodinggradients/ragas/pull/171
Docs: added contents by @shahules786 in https://github.com/explodinggradients/ragas/pull/175
Answer correctness metrics by @shahules786 in https://github.com/explodinggradients/ragas/pull/161
added imports to doc by @shahules786 in https://github.com/explodinggradients/ragas/pull/178
docs: added github and discord links by @jjmachan in https://github.com/explodinggradients/ragas/pull/179
Fix typos by @pitmonticone in https://github.com/explodinggradients/ragas/pull/180
docs: add refferences by @jjmachan in https://github.com/explodinggradients/ragas/pull/177
docs: support notebooks by @jjmachan in https://github.com/explodinggradients/ragas/pull/182
added emojis to docs by @shahules786 in https://github.com/explodinggradients/ragas/pull/183
Docs improvements by @shahules786 in https://github.com/explodinggradients/ragas/pull/186
testset generation: bug fixes by @shahules786 in https://github.com/explodinggradients/ragas/pull/185

New Contributors

@monstertruck made their first contribution in https://github.com/explodinggradients/ragas/pull/168
@pitmonticone made their first contribution in https://github.com/explodinggradients/ragas/pull/180

Full Changelog: https://github.com/explodinggradients/ragas/compare/v0.0.16...v0.0.17

Open source →

0.0.1628 Sept 2023

docs: azure openai quickstart by @jjmachan in https://github.com/explodinggradients/ragas/pull/155

What's Changed

docs: azure openai quickstart by @jjmachan in https://github.com/explodinggradients/ragas/pull/155
fix: llamaIndex issues by @jjmachan in https://github.com/explodinggradients/ragas/pull/156
feat: changing context_relevancy to context_precision by @jjmachan in https://github.com/explodinggradients/ragas/pull/157
Test set generation paradigm by @shahules786 in https://github.com/explodinggradients/ragas/pull/150
Testset generation: documentation by @shahules786 in https://github.com/explodinggradients/ragas/pull/158

Full Changelog: https://github.com/explodinggradients/ragas/compare/v0.0.15...v0.0.16

Open source →

0.0.1525 Sept 2023

added contact info by @shahules786 in https://github.com/explodinggradients/ragas/pull/142

What's Changed

added contact info by @shahules786 in https://github.com/explodinggradients/ragas/pull/142
Add note about when ground truth labels are required by @tleyden in https://github.com/explodinggradients/ragas/pull/146
bug fix: embeddings -> embedding by @starrywheat in https://github.com/explodinggradients/ragas/pull/134
ZeroDivisionError in context_relevance by @DevanshuBrahmbhatt in https://github.com/explodinggradients/ragas/pull/154

New Contributors

@tleyden made their first contribution in https://github.com/explodinggradients/ragas/pull/146
@DevanshuBrahmbhatt made their first contribution in https://github.com/explodinggradients/ragas/pull/154

Full Changelog: https://github.com/explodinggradients/ragas/compare/v0.0.14...v0.0.15

Open source →

0.0.1415 Sept 2023

Update README.md by @emilesilvis in https://github.com/explodinggradients/ragas/pull/139

What's Changed

Update README.md by @emilesilvis in https://github.com/explodinggradients/ragas/pull/139
Fix remap_column_names by @yujonglee in https://github.com/explodinggradients/ragas/pull/140

New Contributors

@emilesilvis made their first contribution in https://github.com/explodinggradients/ragas/pull/139
@yujonglee made their first contribution in https://github.com/explodinggradients/ragas/pull/140

Full Changelog: https://github.com/explodinggradients/ragas/compare/v0.0.13...v0.0.14

Open source →

0.0.1315 Sept 2023

General bug fixes and reliability improvements

What's Changed

fix: spelling of context_relevancy by @jjmachan in https://github.com/explodinggradients/ragas/pull/113
update tutorial notebook by @Yongtae723 in https://github.com/explodinggradients/ragas/pull/125
Fix/column-map: fix bug in column_map argument of evaluation by @jjmachan in https://github.com/explodinggradients/ragas/pull/121
Update README with explanation of Ragas metrics by @Yongtae723 in https://github.com/explodinggradients/ragas/pull/124
Improve metrics by @shahules786 in https://github.com/explodinggradients/ragas/pull/131
add custom embeddings for answer relevance by @starrywheat in https://github.com/explodinggradients/ragas/pull/123
fix: remove dependency on protobuff by @jjmachan in https://github.com/explodinggradients/ragas/pull/132
fix: added support for azure LLM by @jjmachan in https://github.com/explodinggradients/ragas/pull/133
fix: update quickstart notebook by @jjmachan in https://github.com/explodinggradients/ragas/pull/137
fix: linting issues by @jjmachan in https://github.com/explodinggradients/ragas/pull/138

New Contributors

@Yongtae723 made their first contribution in https://github.com/explodinggradients/ragas/pull/125
@starrywheat made their first contribution in https://github.com/explodinggradients/ragas/pull/123

Full Changelog: https://github.com/explodinggradients/ragas/compare/v0.0.12...v0.0.13

Open source →

0.0.126 Sept 2023

chore: added note about consultation by @jjmachan in https://github.com/explodinggradients/ragas/pull/98

What's Changed

chore: added note about consultation by @jjmachan in https://github.com/explodinggradients/ragas/pull/98
llamaIndex: fix the bugs by @jjmachan in https://github.com/explodinggradients/ragas/pull/100
fix typos by @shahules786 in https://github.com/explodinggradients/ragas/pull/102
Limit context relevancy to 1 by @shahules786 in https://github.com/explodinggradients/ragas/pull/105
Improve context relevancy by @shahules786 in https://github.com/explodinggradients/ragas/pull/112

Full Changelog: https://github.com/explodinggradients/ragas/compare/v0.0.11...v0.0.12

Open source →

0.0.1124 Aug 2023

docs: correct docs for answer relevancy by @jjmachan in https://github.com/explodinggradients/ragas/pull/86

What's Changed

docs: correct docs for answer relevancy by @jjmachan in https://github.com/explodinggradients/ragas/pull/86
docs: notebook to showcase langsmith integration by @jjmachan in https://github.com/explodinggradients/ragas/pull/84
Bug fixes and improvements by @shahules786 in https://github.com/explodinggradients/ragas/pull/89
fix(spelling): relavancy -> relevancy by @mikeldking in https://github.com/explodinggradients/ragas/pull/93
feat: add support for langchain by @jjmachan in https://github.com/explodinggradients/ragas/pull/94
feat: column mapping to select columns by @jjmachan in https://github.com/explodinggradients/ragas/pull/91
Context Recall by @shahules786 in https://github.com/explodinggradients/ragas/pull/96

New Contributors

@mikeldking made their first contribution in https://github.com/explodinggradients/ragas/pull/93

Full Changelog: https://github.com/explodinggradients/ragas/compare/v0.0.10...v0.0.11

Open source →

0.0.102 Aug 2023

feat: llama_index integration by @jjmachan in https://github.com/explodinggradients/ragas/pull/78

Main

feat: llama_index integration by @jjmachan in https://github.com/explodinggradients/ragas/pull/78
New Answer relevancy metrics by @shahules786 in https://github.com/explodinggradients/ragas/pull/77
feat(context managers): added Context Managers to help with tracing by @jjmachan in https://github.com/explodinggradients/ragas/pull/83

What's Changed

feat: llama_index integration by @jjmachan in https://github.com/explodinggradients/ragas/pull/78
New Answer relevancy metrics by @shahules786 in https://github.com/explodinggradients/ragas/pull/77
added improved NLI prompt by @shahules786 in https://github.com/explodinggradients/ragas/pull/81
Update README.md by @shahules786 in https://github.com/explodinggradients/ragas/pull/82
feat(context managers): added Context Managers to help with tracing by @jjmachan in https://github.com/explodinggradients/ragas/pull/83
docs: notebook for langsmith integration by @jjmachan in https://github.com/explodinggradients/ragas/pull/85

Full Changelog: https://github.com/explodinggradients/ragas/compare/v0.0.9...v0.0.10

Open source →

0.0.927 Jul 2023

fix: validation was incorrect by @jjmachan in https://github.com/explodinggradients/ragas/pull/75

patch release for v0.0.8

What's Changed

fix: validation was incorrect by @jjmachan in https://github.com/explodinggradients/ragas/pull/75
chore: added an e2e test in Makefile by @jjmachan in https://github.com/explodinggradients/ragas/pull/76

Full Changelog: https://github.com/explodinggradients/ragas/compare/v0.0.8...v0.0.9

Open source →

0.0.827 Jul 2023withdrawn: bug in the code

Critique metrics by @shahules786 in https://github.com/explodinggradients/ragas/pull/70

Main

Critique metrics by @shahules786 in https://github.com/explodinggradients/ragas/pull/70

What's Changed

fix: created new class for MetricWithLLM by @jjmachan in https://github.com/explodinggradients/ragas/pull/71
chore: support unit testing on python 3.11 by @jjmachan in https://github.com/explodinggradients/ragas/pull/69
Critique metrics by @shahules786 in https://github.com/explodinggradients/ragas/pull/70
feat: add validation step by @jjmachan in https://github.com/explodinggradients/ragas/pull/72
fix: n_swapped check for generate by @jjmachan in https://github.com/explodinggradients/ragas/pull/73

Full Changelog: https://github.com/explodinggradients/ragas/compare/0.0.7...v0.0.8

Open source →

0.0.720 Jul 2023

feat: added langchains LLM by @jjmachan in https://github.com/explodinggradients/ragas/pull/66

What's Changed

feat: added langchains LLM by @jjmachan in https://github.com/explodinggradients/ragas/pull/66
docs: notebook guide for custom llm by @jjmachan in https://github.com/explodinggradients/ragas/pull/68

Full Changelog: https://github.com/explodinggradients/ragas/compare/0.0.6...0.0.7

Open source →

0.0.615 Jul 2023

Context Relevancy v2 - measures how relevant is the retrieved context to the prompt. This is done using a combination of OpenAI models and cross-encod

Main

Context Relevancy v2 - measures how relevant is the retrieved context to the prompt. This is done using a combination of OpenAI models and cross-encoder models. To improve the score one can try to optimize the amount of information present in the retrieved context.

What's Changed

added analytics by @jjmachan in https://github.com/explodinggradients/ragas/pull/58
Context Relevancy v2 by @shahules786 in https://github.com/explodinggradients/ragas/pull/59
doc: added numpy style documentation to context_relavency by @jjmachan in https://github.com/explodinggradients/ragas/pull/62
updated docs by @shahules786 in https://github.com/explodinggradients/ragas/pull/64
fix: error in handling device for tensors by @jjmachan in https://github.com/explodinggradients/ragas/pull/61
chore: renamed files and added tqdm by @jjmachan in https://github.com/explodinggradients/ragas/pull/65

Full Changelog: https://github.com/explodinggradients/ragas/compare/0.0.5...0.0.6

Open source →

0.0.510 Jul 2023

chore: removed lfs files by @jjmachan in https://github.com/explodinggradients/ragas/pull/56

What's Changed

chore: removed lfs files by @jjmachan in https://github.com/explodinggradients/ragas/pull/56
fix: Result output by @jjmachan in https://github.com/explodinggradients/ragas/pull/57

Full Changelog: https://github.com/explodinggradients/ragas/compare/0.0.4...0.0.5

Open source →

0.0.410 Jul 2023

Rename metrics by @shahules786 in https://github.com/explodinggradients/ragas/pull/48

Important feats

Rename metrics by @shahules786 in https://github.com/explodinggradients/ragas/pull/48
feat: open usage tracking by @jjmachan in https://github.com/explodinggradients/ragas/pull/52

What's Changed

Update README.md by @jjmachan in https://github.com/explodinggradients/ragas/pull/42
Hotfix: Update Readme [ SpellingError ] by @MANISH007700 in https://github.com/explodinggradients/ragas/pull/43
Update metrics.md by @shahules786 in https://github.com/explodinggradients/ragas/pull/45
added discord server by @jjmachan in https://github.com/explodinggradients/ragas/pull/47
Rename metrics by @shahules786 in https://github.com/explodinggradients/ragas/pull/48
feat: open usage tracking by @jjmachan in https://github.com/explodinggradients/ragas/pull/52
docs: moved quickstart by @jjmachan in https://github.com/explodinggradients/ragas/pull/54
docs: fix quickstart and readme by @jjmachan in https://github.com/explodinggradients/ragas/pull/55

New Contributors

@MANISH007700 made their first contribution in https://github.com/explodinggradients/ragas/pull/43

Full Changelog: https://github.com/explodinggradients/ragas/compare/v0.0.3...0.0.4

Open source →

0.0.39 Jun 2023

v0.0.3 is a major design change

We have added 3 new metrics that help you answer how factually correct is your generated answers, how relevant are the answers to the question and how relevant are the contexts returned form the retriever to the questions. This gives you a sense of the performance of both you generation and retrieval steps. We also have a "ragas_score" which is unified score to give a single metric about your pipelines.

checkout the quickstart to see how it works: https://github.com/explodinggradients/ragas/blob/main/examples/quickstart.ipynb

Open source →

0.0.3rc19 Jun 2023pre-release

Nothing published for this version

0.0.223 May 2023

Update README by @shahules786 in https://github.com/explodinggradients/ragas/pull/27

What's Changed

Update README by @shahules786 in https://github.com/explodinggradients/ragas/pull/27
Added evaluation benchmark by @shahules786 in https://github.com/explodinggradients/ragas/pull/29
Fix bert score by @shahules786 in https://github.com/explodinggradients/ragas/pull/30
Update Readme by @shahules786 in https://github.com/explodinggradients/ragas/pull/31
fix: lazyloading of model used in metrics to speed up import by @jjmachan in https://github.com/explodinggradients/ragas/pull/32

Full Changelog: https://github.com/explodinggradients/ragas/compare/0.0.1...0.0.2

Open source →

0.0.114 May 2023

fix: added a spacy check before installation by @jjmachan in https://github.com/explodinggradients/ragas/pull/22

MVP release!!!

What's Changed

fix: added a spacy check before installation by @jjmachan in https://github.com/explodinggradients/ragas/pull/22
fix: remove incorrect version by @jjmachan in https://github.com/explodinggradients/ragas/pull/23
Update Metrics info and minor bug fixes by @shahules786 in https://github.com/explodinggradients/ragas/pull/24
Update README by @shahules786 in https://github.com/explodinggradients/ragas/pull/25
fix: some fixes and readme by @jjmachan in https://github.com/explodinggradients/ragas/pull/26

Full Changelog: https://github.com/explodinggradients/ragas/compare/0.0.1a6...0.0.1

Open source →

0.0.1a714 May 2023pre-release

Nothing published for this version

0.0.1a614 May 2023pre-release

chore: fix issues with released package by @jjmachan in https://github.com/explodinggradients/ragas/pull/20

What's Changed

chore: fix issues with released package by @jjmachan in https://github.com/explodinggradients/ragas/pull/20
Initialise all metics by @shahules786 in https://github.com/explodinggradients/ragas/pull/21

Full Changelog: https://github.com/explodinggradients/ragas/compare/0.0.1a5...0.0.1a6

Open source →

0.0.1a514 May 2023pre-release

chore: mention release environment by @jjmachan in https://github.com/explodinggradients/ragas/pull/19

testing release workflow

What's Changed

chore: mention release environment by @jjmachan in https://github.com/explodinggradients/ragas/pull/19

Full Changelog: https://github.com/explodinggradients/ragas/compare/0.0.1a4...0.0.1a5

Open source →

Your coding agent can read these notes before it upgrades. Set up the MCP server →