evidently

What's Changed

• Remove explicit pyarrow dep by @mike0sv in https://github.com/evidentlyai/evidently/pull/885
• New service API by @mike0sv in https://github.com/evidentlyai/evidently/pull/884
• Docs for 0.4.10 by @elenasamuylova in https://github.com/evidentlyai/evidently/pull/883
• An example of data drift detection with feature importances by @emeli-dral in https://github.com/evidentlyai/evidently/pull/889
• Version 0.4.11 by @emeli-dral in https://github.com/evidentlyai/evidently/pull/890

Full Changelog: https://github.com/evidentlyai/evidently/compare/v0.4.10...v0.4.11

What's Changed

• Docs by @elenasamuylova in https://github.com/evidentlyai/evidently/pull/857
• Fix docs: replace --demo-project flag with --demo-projects all by @DimaAmega in https://github.com/evidentlyai/evidently/pull/858
• fix/return special json parser by @DimaAmega in https://github.com/evidentlyai/evidently/pull/860
• Add better json encoder to collector by @mike0sv in https://github.com/evidentlyai/evidently/pull/868
• refactor/dashboard by @DimaAmega in https://github.com/evidentlyai/evidently/pull/863
• UI: playwright by @DimaAmega in https://github.com/evidentlyai/evidently/pull/864
• Add metrics by @0lgaF in https://github.com/evidentlyai/evidently/pull/871
• Update workflows by @DimaAmega in https://github.com/evidentlyai/evidently/pull/875
• Add fi test by @0lgaF in https://github.com/evidentlyai/evidently/pull/876
• Add RowCountTrigger to Collector Service by @magdalenakuhn17 in https://github.com/evidentlyai/evidently/pull/859
• fix: Fix reference data plot for classification probability distribution by @nedwebster in https://github.com/evidentlyai/evidently/pull/866
• Add additional data to metric preset generation by @mike0sv in https://github.com/evidentlyai/evidently/pull/877
• Fix vis for spark df with timestamp by @mike0sv in https://github.com/evidentlyai/evidently/pull/865
• feature/filter by metadata by @DimaAmega in https://github.com/evidentlyai/evidently/pull/873
• Add preset by @0lgaF in https://github.com/evidentlyai/evidently/pull/878
• fix/nivo by @DimaAmega in https://github.com/evidentlyai/evidently/pull/879
• Fix empty rows when paging by @DimaAmega in https://github.com/evidentlyai/evidently/pull/880

New Contributors

• @nedwebster made their first contribution in https://github.com/evidentlyai/evidently/pull/866

Full Changelog: https://github.com/evidentlyai/evidently/compare/v0.4.9...v0.4.10

What's Changed

• add some return type hints by @jameslamb in https://github.com/evidentlyai/evidently/pull/834
• Fix two FutureWarnings maintaining current behaviour by @aaalexlit in https://github.com/evidentlyai/evidently/pull/842
• Fix the bug by @0lgaF in https://github.com/evidentlyai/evidently/pull/845
• Fix to draw scatter plots by @mike0sv in https://github.com/evidentlyai/evidently/pull/849
• Feature/fsspec workspace by @mike0sv in https://github.com/evidentlyai/evidently/pull/850
• feat/setup evidently-ui-lib ts package + a few UI changes by @DimaAmega in https://github.com/evidentlyai/evidently/pull/847
• UI: fix readme files since UI changes by @DimaAmega in https://github.com/evidentlyai/evidently/pull/853
• Add metrics by @0lgaF in https://github.com/evidentlyai/evidently/pull/841
• Support file-like objects for report.save_html by @mike0sv in https://github.com/evidentlyai/evidently/pull/854
• Add feature importance by @0lgaF in https://github.com/evidentlyai/evidently/pull/852
• Counter panels for TestSuites by @mike0sv in https://github.com/evidentlyai/evidently/pull/843

New Contributors

• @aaalexlit made their first contribution in https://github.com/evidentlyai/evidently/pull/842

Full Changelog: https://github.com/evidentlyai/evidently/compare/v0.4.8...v0.4.9

Updates

• feature/ui update dependencies by @DimaAmega in https://github.com/evidentlyai/evidently/pull/818
• Add test suite dashboard panels by @mike0sv in https://github.com/evidentlyai/evidently/pull/817
• Spark engine by @mike0sv in https://github.com/evidentlyai/evidently/pull/806
• Better CI by @DimaAmega in https://github.com/evidentlyai/evidently/pull/822
• Doc: dvclive integration by @francesco086 in https://github.com/evidentlyai/evidently/pull/825
• Additional datasets field for run by @mike0sv in https://github.com/evidentlyai/evidently/pull/824
• MobAd BigData Beeline Kazakhstan: add lift table and lift curve metrics by @IlmuratTokhtakhunov in https://github.com/evidentlyai/evidently/pull/544
• Adult dataset demo by @mike0sv in https://github.com/evidentlyai/evidently/pull/837
• Add spark example by @mike0sv in https://github.com/evidentlyai/evidently/pull/838

Fixes:

• Fix RemoteWorkspace verify by @vlin-lgtm in https://github.com/evidentlyai/evidently/pull/823
• refactor: Rewrote verify method of RemoteWorkspace by @mou in https://github.com/evidentlyai/evidently/pull/827
• [docs] fix typos in docs by @jameslamb in https://github.com/evidentlyai/evidently/pull/828
• Fix after merge 818 by @DimaAmega in https://github.com/evidentlyai/evidently/pull/830
• fix lint by @mike0sv in https://github.com/evidentlyai/evidently/pull/831
• fix for lift metrics by @emeli-dral in https://github.com/evidentlyai/evidently/pull/835

Full Changelog: https://github.com/evidentlyai/evidently/compare/v0.4.7...v0.4.8

Updates:

• Update dataset_correlations_metric.py by @VishalKumar-S in https://github.com/evidentlyai/evidently/pull/814

Fixes:

• Fix_typo by @0lgaF in https://github.com/evidentlyai/evidently/pull/812
• Deprecate _as_pandas by @mike0sv in https://github.com/evidentlyai/evidently/pull/816
• Respect tags for counter panels by @mike0sv in https://github.com/evidentlyai/evidently/pull/821

Full Changelog: https://github.com/evidentlyai/evidently/compare/v0.4.6...v0.4.7

Updates: #775 #777 #778 #787 #788 #789 #790 #791 #796 #797 #798 #803

Fixes: #769 #776 #774 #792 #801 #808

Updates: #746 #751 #755 #759 #762

Fixes: #750 #753 #758 #760 #761 #764 #765

Updates: #734

Fixes: #743 #744

Updates: #720 #727 #728 #729 #730

Fixes: #721 #723

Updates: #711 #714

Fixes: #681 #692 #695 #696 #698 #704 #707 #712 #713

Updates: #631 #634 #633 #668 #672 #677 #684 #687 #690

Fixes: #641 #651 #665

Breaking Changes: #644

Updates: #619 #622 #628 #629

Experimental: #600

Breaking Changes: Renamed parameter lemmatisize -> lemmatize in TriggerWordsPresence text descriptor

Fixes: #607 #611 #613 #615 #618 #617 #623 #624 #625 #626 #630

Breaking Changes: #601 Since this version, the following metrics will render aggregated plots instead of raw data plots:

• ColumnDriftMetric
• DataDriftTable
• ColumnValuePlot
• TextDescriptorsDriftMetric
• ColumnSummaryMetric
• RegressionPredictedVsActualScatter
• RegressionPredictedVsActualPlot
• RegressionErrorPlot
• RegressionAbsPercentageErrorPlot
• RegressionErrorNormality
• RegressionTopErrorMetric
• ClassificationClassSeparationPlot
• ClassificationQualityByFeatureTable
• RegressionErrorBiasTable
• TargetByFeaturesTable

To return raw data plots use render option :

• On a metric level:

report = Report(
	metrics=[
	RegressionPredictedVsActualScatter(options={"render": {"raw_data": True}}),
])

• On a report level:

report = Report(
    metrics=[
      RegressionQualityMetric(),
      RegressionPredictedVsActualScatter(),
    ],
    options={"render": {"raw_data": True}}
  )

Updates: #588 #589 #597

Fixes: #593

Breaking Changes: #585 Drift detection method text_content_drift was renamed to perc_text_content_drift This happened because an additional text data drift detection method was added: abs_text_content_drift. This drift detection technique works more efficiently with large datasets.

Updates: #562 #564 #566 #567 #569 #570 #587

Fixes: #568 #572 #573 #574 #575 #579 #580 #582 #583 #586

Breaking Changes: Old API is no longer supported! There are no more Dashboards, Model Profiles, Tabs, and Profile Sections! #550

Updates: JSON output customisation: #543 #545 #548

Text descriptors support in column metrics and tests: #542 #546 #553 #557

Embeddings Drift Metric #558

Fixes: #560

New Examples: How to customize JSON output for Reports and TestSuites? How to calculate drift for embeddings? How to apply column metrics for text descriptors?

Updates: #517 #533

Streamlit Integration Example: #531

Fixes: #523 #528 #529 #532 #534 #535

Updates:

1. Added text descriptor TriggerWordsPresence
2. Added an option to specify which text descriptors should be used in the following metrics:

• TextDescriptorsDriftMetric
• TextDescriptorsDistribution
• TextDescriptorsCorrelationMetric

1. Added an option to specify which text descriptors should be used in the TextOverviewPreset

Fixes: #512 #519 #520 #521 #522

Updates: Instead of using just Pearson correlation, the following tests automatically choose the correlation method depending on target and prediction types:

• TestTargetPredictionCorrelation
• TestTargetFeaturesCorrelations
• TestPredictionFeaturesCorrelations

Fixes: #446 #499 #504 #505 #511

Updated project build to be compatible with python 3.11

Conda-compatible project build: prebuilt UI is stored in the repo; there is no need to build it locally.

Breaking Changes:

• Python 3.6 is no longer supported

Updates:

• New parameter “text_features” was added to ColumnMapping. text_features parameter takes a list with feature names: “column_mapping.text_features=[’text_feature_1’, ‘text_feature_2’, …, ‘text_feature_k’]”
• The following metrics now support text features:
  • DatasetSummaryMetric()
  • DatasetMissingValuesMetric()
  • ColumnSummaryMetric(column_name="name")
  • ColumnMissingValuesMetric(column_name="name")
  • ColumnRegExpMetric(column_name="name", reg_exp=r".ticket.")
  • ConflictPredictionMetric()
  • ConflictTargetMetric()
  • DatasetCorrelationsMetric()
  • DatasetDriftMetric()
  • DataDriftTable()
  • ColumnDriftMetric(column_name=”name”)
  • TargetByFeaturesTable(columns=columns)
  • ClassificationQualityByFeatureTable()
  • RegressionErrorBiasTable()
• All metric presets now support text features
• All tests based on metrics that support text features also support texts
• The following test presets now support text features:
  • NoTargetPerformanceTestPreset
  • DataStabilityTestPreset
  • DataQualityTestPreset
  • DataDriftTestPreset
• Added metric TextDescriptorsDriftMetric for text data
• Added metric TextDescriptorsDistribution for text data
• Added metric TextDescriptorsCorrelationMetric for text data
• Added TextOverviewPreset(column_name=”name”) for text data. The preset includes:
  • ColumnSummaryMetric
  • TextDescriptorsDistribution
  • TextDescriptorsCorrelation
  • ColumnDriftMetric (if reference dataset is provided)
  • TextDescriptorsDriftMetric (if reference dataset is provided)

Changes:

• Method get_parameters(self) -> Optional[tuple] from Metric(Generic[TResult]) class was updated and became optional. The algorithm to determine metric parameters and create a tuple with metric parameters and its values is updated to cover more parameter types. Since the algorithm has been significantly updated, get_parameters method should be implemented in a custom metric class only if specific behaviour is needed. For most standard scenarios, the default version is sufficient.
• Metric deduplication now includes not only metric calculation parameters but features as well. This helps to deduplicate calculations of the same metrics on top of the same features and as a result, reduce calculation time.

Fixes:

• #484
• #490

Updates: Added Maximum-Mean-Discrepancy (MMD) test #383 Added an example of the integration with Metaflow #468

Fixes: #463 #471 #472 #473 #474 #476

Breaking Changes:

NOTE: Dashboards, Profiles, Tabs and Profile Sections are now DEPRECATED and will be completely REMOVED in the nearest releases.

Deleted NumTargetDriftPreset (use TargetDriftPreset instead) Deleted CatTargetDriftPreset (use TargetDriftPreset instead)

Renamed Parameters:

• classification_threshold -> probas_threshold
  this afects: ClassificationQualityMetric , TestAccuracyScore, TestPrecisionScore, TestRecallScore, TestF1Score, TestTPR, TestTNR, TestFPR, TestFNR, TestPrecisionByClass, TestRecallByClass, TestF1ByClass, ClassificationPreset, BinaryClassificationTestPreset

• threshold-> stattest_threshold this afects: ColumnDriftMetric, TestColumnValueDrift, BinaryClassificationTestPreset, BinaryClassificationTopKTestPreset, MulticlassClassificationTestPreset

• all_features_stattest  -> stattest & all_features_threshold -> stattest_threshold this afects: DataDriftTable, DatasetDriftMetric, TestNumberOfDriftedColumns, TestShareOfDriftedColumns, DataDriftPreset, TargetDriftPreset, DataDriftTestPreset, NoTargetPerformanceTestPreset

• cat_features_stattest -> cat_stattest  & cat_features_threshold -> cat_stattest_threshold  this afects: DataDriftTable, DatasetDriftMetric, TestNumberOfDriftedColumns, TestShareOfDriftedColumns, DataDriftPreset, TargetDriftPreset, DataDriftTestPreset, NoTargetPerformanceTestPreset

• num_features_stattest -> num_stattest & num_features_stattest -> num_stattest_threshold  this afects: DataDriftTable, DatasetDriftMetric, TestNumberOfDriftedColumns, TestShareOfDriftedColumns, DataDriftPreset, TargetDriftPreset, DataDriftTestPreset, NoTargetPerformanceTestPreset

• per_feature_stattest -> per_column_stattest & per_feature_stattest -> per_column_stattest_threshold this afects: DataDriftTable, DatasetDriftMetric, TestNumberOfDriftedColumns, TestShareOfDriftedColumns, DataDriftPreset, TargetDriftPreset, DataDriftTestPreset, NoTargetPerformanceTestPreset

Renamed Tests:

• TestColumnValueDrift  -> TestColumnDrift
• TestColumnValueRegExp -> TestColumnRegExp
• TestValueQuantile -> TestColumnQuantile

Updates:

Added top_error parameter to RegressionErrorBiasTable metric #422 Added ClassificationDummyMetric metric #445 Added RegressionDummyMetric metric #445 Added ConflictPredictionMetric metric #455 Added ConflictTargetMetric metric #455

Added API reference DRAFT https://docs.evidentlyai.com/reference/api-reference

Added new Statistical Tests:

• Apps-Singleton test #363
• Total Variation Distance (TVD) #391

Fixes:

#431 #438 #451 #458

Breaking Changes: Metrics Rename: ClassificationQuality -> ClassificationQualityMetric ProbabilityDistribution -> ClassificationProbDistribution

Tests Rename: TestHighlyCorrelatedFeatures -> TestHighlyCorrelatedColumns TestFeatureValueMin -> TestColumnValueMin TestFeatureValueMax -> TestColumnValueMax TestFeatureValueMean -> TestColumnValueMean TestFeatureValueMedian -> TestColumnValueMedian TestFeatureValueStd -> TestColumnValueStd TestNumberOfDriftedFeatures -> TestNumberOfDriftedColumns TestShareOfDriftedFeatures -> TestShareOfDriftedColumns TestFeatureValueDrift -> TestColumnValueDrift

Updates: #400 #373

Fixes: #371 #421 #419 #418 #417 #416 #415

Breaking Changes:

All Test Presets were renamed.
TestPreset suffix was added to original names:

• NoTargetPerformance -> NoTargetPerformanceTestPreset
• DataQuality -> DataQualityTestPreset
• DataStability -> DataStabilityTestPreset
• DataDrift -> DataDriftTestPreset
• Regression -> RegressionTestPreset
• MulticlassClassification -> MulticlassClassificationTestPreset
• BinaryClassificationTopK -> BinaryClassificationTopKTestPreset
• BinaryClassification -> BinaryClassificationTestPreset

Updates:

Added DataDrift metrics:

• DatasetDriftMetric
• DataDriftTable
• ColumnValuePlot
• TargetByFeaturesTable

Added DataQuality metrics:

• ColumnDistributionMetric
• ColumnQuantileMetric
• ColumnCorrelationsMetric
• ColumnValueListMetric
• ColumnValueRangeMetric
• DatasetCorrelationsMetric

Added DataIntegrity metrics:

• ColumnSummaryMetric
• ColumnMissingValuesMetric
• DatasetSummaryMetric
• DatasetMissingValuesMetric

Added Classification metrics:

• ClassificationQuality
• ClassificationClassBalance
• ClassificationConfusionMatrix
• ClassificationQualityByClass
• ClassificationClassSeparationPlot
• ProbabilityDistribution
• ClassificationRocCurve
• ClassificationPRCurve
• ClassificationPRTable
• ClassificationQualityByFeatureTable

Added Regression metrics:

• RegressionQualityMetric
• RegressionPredictedVsActualScatter
• RegressionPredictedVsActualPlot
• RegressionErrorPlot
• RegressionAbsPercentageErrorPlot
• RegressionErrorDistribution
• RegressionErrorNormality
• RegressionTopErrorMetric
• RegressionErrorBiasTable

Added MetricPresets:

• DataDriftPreset
• DataQualityPreset
• RegressionPreset
• ClassificationPreset

Added New Statistical Tests

• Anderson-Darling test for numerical features
• Cramer Von Mises test for numerical features
• Hellinger distance test for numerical and categorical features
• Mann-Whitney U-rank test for numerical features
• Cressie-Read power divergence test for categorical features

Fixes: #334 #353 #361 #367

Updates:

• Replaced BaseWidgetInfo with helpers: https://github.com/evidentlyai/evidently/pull/326
• Added metrics generator for column-based metrics: https://github.com/evidentlyai/evidently/pull/323
• Added black and isort: https://github.com/evidentlyai/evidently/pull/322

Fixes:

• https://github.com/evidentlyai/evidently/pull/340
• https://github.com/evidentlyai/evidently/pull/341
• https://github.com/evidentlyai/evidently/pull/336

Updates:

• Introduced Report - an object, that unites Dashboard and Profile functionality
• Introduced MetricPreset - an object, that replaces Tab and ProfileSection
• Implemented following MetricPresets: DataDrift, DataQuality (limited content), CatTargetDrift, NumTargetDrift, RegressionPerformance, ClassificationPerformance

Fixes:

• #312

Updates:

• Implemented function generate_column_tests() to generate similar tests for many columns automatically

Dataset Null-related tests

• Implemented TestNumberOfNulls to replace TestNumberOfNANs and TestNumberOfNullValues
• Implemented TestShareOfNulls
• Implemented TestShareOfColumnsWithNulls
• Implemented TestShareOfRowsWithNulls
• Implemented TestNumberOfDifferentNulls

Column Null-related tests

• Implemented TestColumnNumberOfNulls to replace TestColumnNumberOfNullValues
• Implemented TestColumnShareOfNulls to replace TestColumnNANShare

Fixes:

• Fixed metric duplication to reduce an amount of calculations while building TestSuites (basically, now same metrics from the one test suite are not recalculated multiple times)
• Implemented NAN filtering for all dashboards in way, that each column is filtered separately

Updates:

• added TPR, TNR, FPR, FNR Tests for Binary Classification Model Performance
• Renamed status "No Group" to "Dataset-level tests" in TestSuites filtering menu

Fixes:

• #207
• #265
• #256
• fixed unit tests for different versions of python and pandas

Updates:

1. Updated the UI to let users group tests by the following properties:

• All tests
• By Status
• By feature
• By test type
• By test group

1. New Tests:

• Added tests for binary probabilistic classification models
• Added tests for multiclass classification models
• Added tests for multiclass probabilistic classification models The full list of tests will be available in the docs.

1. New Tests Presets:

• Regression
• MulticlassClassification
• BinaryClassificationTopK
• BinaryClassification

• added default configurations for Data Quality Tests
• added default configurations for Data Integrity Tests
• added visualisation for Data Quality Tests
• added visualisation for Data Integrity Tests
• Test descriptions are updated (column names are highlighted)

Implemented new interfaces to test data and models in a batch: Test Suite.

Implemented the following Individual tests:

• TestNumberOfColumns()
• TestNumberOfRows()
• TestColumnNANShare()
• TestShareOfOutRangeValues()
• TestNumberOfOutListValues()
• TestMeanInNSigmas()
• TestMostCommonValueShare()
• TestNumberOfConstantColumns()
• TestNumberOfDuplicatedColumns()
• TestNumberOfDuplicatedRows()
• TestHighlyCorrelatedFeatures()
• TestTargetFeaturesCorrelations()
• TestShareOfDriftedFeatures()
• TestValueDrfit()
• TestColumnsType()

Implemented the following test presets:

• Data Quality. This preset is focused on the data quality issues like duplicate rows or null values.
• Data Stability. This preset identifies the changes in the data or differences between the batches.
• Data Drift. This one compares feature distributions using statistical tests and distance metrics.
• NoTargetPerformance. This preset combines several checks to run when there are model predictions, there are no actuals or ground truth labels. This includes checking for prediction drift and some of the data quality and stability checks.

Updates:

• Updated DataDriftTab: added target and prediction rows in DataDrift Table widget
• Updated CatTargetDriftTab: added additional widgets for probabilistic cases in both binary and multiclasss probabilistic classification, particularly widget for label drift and class probability distributions.

Fixes:

• #233
• fixed previes in DataDrift Table widget. Now histogram previews for refernce and current data share an x-axis. This means that bins order in refernce and current histograms is the same, it makes visual distribution comparion esier.

Release scope:

1. Stat test auto selection algorithm update: https://docs.evidentlyai.com/reports/data-drift#how-it-works

For small data with <= 1000 observations in the reference dataset:

• For numerical features (n_unique > 5): two-sample Kolmogorov-Smirnov test.
• For categorical features or numerical features with n_unique <= 5: chi-squared test.
• For binary categorical features (n_unique <= 2), we use the proportion difference test for independent samples based on Z-score. All tests use a 0.95 confidence level by default.

For larger data with > 1000 observations in the reference dataset:

• For numerical features (n_unique > 5): Wasserstein Distance.
• For categorical features or numerical with n_unique <= 5): Jensen–Shannon divergence. All tests use a threshold = 0.1 by default.

1. Added options for setting custom statistical test for Categorical and Numerical Target Drift Dashboard/Profile: cat_target_stattest_func: Defines a custom statistical test to detect target drift in CatTargetDrift. num_target_stattest_func: Defines a custom statistical test to detect target drift in NumTargetDrift.

2. Added options for setting custom threshold for drift detection for Categorical and Numerical Target Drift Dashboard/Profile: cat_target_threshold: Optional[float] = None num_target_threshold: Optional[float] = None These thresholds highly depends on selected stattest, generally it is either threshold for p_value or threshold for a distance.

Fixes:
#207

StatTests The following statistical tests now can be used for both numerical and categorical features:

• 'jensenshannon'
• 'kl_div'
• 'psi

Grafana monitoring example

• Updated the example to be used with several ML models
• Added monitors for NumTargetDrift, CatTargetDrift

Colour Scheme Support for custom colours in the Dashboards:

• primary_color
• secondary_color
• current_data_color
• reference_data_color
• color_sequence
• fill_color
• zero_line_color
• non_visible_color
• underestimation_color
• overestimation_color
• majority_color

Statistical Test: Support for user implemented statistical tests Support for custom statistical tests in Dashboards and Profiles Available tests:

• 'ks'
• 'z'
• 'chisquare'
• 'jensenshannon'
• 'kl_div'
• 'psi'
• 'wasserstein' more info: docs

Fixes: #193

Custom Text Comments in Dashboards

• Added type type="text” for BaseWidgetInfo (for text widgets implementation)
• Markdown syntax is supported

see the example: https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/text_widget_usage_iris.ipynb

• Data Quality Dashboard: add dataset overview widget
• Data Quality Dashboard: add correlations widget
• Speeded uploading via preview plots optimisation
• Paging in Data Quality feature table widget

• DataQualityTab() is now available for Dashboards! The Tab helps to observe data columns, explore their properties and compare datasets.
• DataQualityProfileSection() is available for Profiles as well.
• ColumnMapping update: added task parameter to specify the type of machine learning problem. This parameter is used by DataQualityAnalyzer and some data quality widgets to calculate metrics and visuals with the respect to the task.

• Added monitors for NumTargetDrift, CatTargetDrift, ClassificationPerformance, ProbClassificationPerformance
• Fixed RegressionPerformance monitor
• Supported strings as a categorical features in DataDrift and CatTargetDrift dashboards
• Supported boolean features in DataDrift dashboard

Analyzers Refactoring: analyzer result became a structured object instead of a dictionary for all Analyzers

The following Quality Metrics Options are added:

• conf_interval_n_sigmas (the width of confidence intervals ): int = DEFAULT_CONF_INTERVAL_SIZE
• classification_treshold (the threshold for true labels): float = DEFAULT_CLASSIFICATION_TRESHOLD
• cut_quantile (cut the data by right, left and two-sided quantiles): Union[None, Tuple[str, float], Dict[str, Tuple[str, float]]] = None

Added backward compatibility for imports:

• Widgets and Tabs can be imported from evidently directly, but this is deprecated behavior and cause warning
• Sections can be imported from evidently directly, but this is deprecated behavior and cause warning

• Library source code is moved to the src/evidently folder
• Docs, Tests, and Examples are moved to the top level of the repo
• Widgets and Tabs are moved inside of the src/evidently/dashboard folder, as those are parts of the Dashboard
• Sections are moved inside of the src/evidently/model_profile folder, as those are parts of the Model_profiles
• Docs are stored in the repo: docs/book folder
• DataDriftAnalyzer refactoring: analyzer results became a structured object instead of a dictionary

• fixed: input DataFrames cannot be changed during any calculations (fixed by making shallow copies)
• fixed: chi-square statistical test uses normalized frequencies (with respect to the latest scipy version)
• current dataset is optional for Performace Tabs and Sections calculation (None value can be passed)
• improved readme

Data Drift Options:

• Created confidence: Union[float, Dict[str, float]] - option can take a float or a dict as an argument. If float has passed, then this confidence level will be used for all features. If dict has passed, then specified features will have a custom confidence levels (all the rest will have default confidence level = 0.95)
• Updated nbinsx: Union[int, Dict[str, int]] - option can take an int or a dict as an argument. If int has passed, then this number of bins will be used for all features. If dict has passed, then specified features will have a custom number of bins (all the rest will have default number of bins = 10)
• Updated feature_stattest_func: Union[None, Callable, Dict[str, Callable]] - option can take a function or a dict as an argument. If a function has passed, then this function will be used to measure drift for all features. If dict has passed, then for specified features custom functions will be used (all the rest features will be processed by an internal algorithm of drift measurement)

Package building:

• Fixed dependencies

• Support widgets order for include_widgets parameter
• Support an ability to add a custom widget to Tabs with include_widgets parameter
• Moved options to a separate module
• Added options to specify statistical tests for DataDrift and TargetDrift Dashboards: stattest_func - to set a custom statistical test for all the features feature_stattest_func - to set a custom statistical tests for each individual feature cat_target_stattest_func - to set a custom statistical test for categorical target num_target_stattest_func - to set a custom statistical test for numerical target
• Refactored Widgets and Tabs for simpler customisation

• Supported custom list of Widgets for Tabs in Dashboard with help of verbose_level and include_widgets parameters
• Added parameter verbose_level: 0 - to create a Tab with the shortest list of Widgets, 1 - to create a full Tab
• Added parameter include_widgets : ["Widget Name 1", "Widget Name 2", etc]. This parameter overwrites verbose_level (if both are specified) and allows to set a custom list of Widgets
• Added Tab.list_widgets() method to list all the available Widgets for the current Tab
• Created Options entity to specify Widgets and Tabs customisable settings
• Created ColumnMapping entity instead column_mapping python dictionary