PyPI · #1765 most downloaded on PyPI

acryl-datahub

1.7.0.145.0M downloads/modatahub-project/datahub ↗New releases in your daily digest, changelog included.

DataHub ingestion framework and CLI — connect, extract, and push metadata from 50+ data sources into your DataHub catalog

Last release 5 days ago
30 Sep 2026
Ships on a steady schedule
a new release about every 8 days
Some releases are documented
notes for 21 of the last 60 stable releases
14 versions withdrawn
withdrawn after publishing
6 years old
1287 releases · first in 2021

Release timeline

1287 releases since 2021

202120222023202420252026

20212026

One column per quarter.

Releases

0.8.32.26 Apr 2022
Nothing published for this version
0.8.32.2rc16 Apr 2022pre-release
Nothing published for this version
0.8.32.15 Apr 2022
Nothing published for this version
0.8.324 Apr 2022
We're excited to announce View-based RBAC Policies! You can now create and apply view-only permissions to your DataHub end-users, providing more robus
Release Highlights

User Experience

We're excited to announce View-based RBAC Policies! You can now create and apply view-only permissions to your DataHub end-users, providing more robust access controls.

We've also included some small (but impactful!) improvements to UX, including:

Display recent search terms when beginning the search flow

Consistently displaying entity subtypes for dbt, Looker, Kafka, & more. Think: Kafka entities are displayed as "topics" instead of "datasets"

Ingestion Highlights

New! Protobuf ingestion (shoutout to @leifker for this Community-led contribution!)

Initial work to support a "Notebook" entity (shoutout to @tc350981 for spearheading this work!!)

Stateful ingestion for dbt is now supported

Ongoing improvements to our Tableau ingestion source from @nandacamargo & @cuong-pham

Improvements to handling database aliases for Redshift ingestion

Improvements to S3 source:

Add containers for datasets

Support platform_instance

Support for folder level datasets

Increased flexibility to specify dataset paths

Ingestion Fixes:

Snowflake Usage - log warning instead of error out & other error handling

Snowflake allow/deny patterns

Examples of allow/deny patterns added to docs

Full Commit Log

#4570 @gabe-lyons fix(search): handle commas in search queries in the UI

#4557 @daha fix: replace direct and indirect references to linkedin with datahub-project

#4569 @dexter-mh-lee fix(policy): Add view entity page priv to all entity types

#4567@anshbansal fix(bigquery): missing dependency

#4548 @mayurinehate fix(tableau): gracefully stop ingestion if tableau sign in not successful

#4564 @dexter-mh-lee fix(docs): fix logo links on ingestion docs

#4396 @Abhiram98 feat(ingestion): schema, table filtering for redshift-usage

#4554 @darapuk (fix): Update path generated when creating LookML URL

#4549 @maggiehays docs: add sumup logo

#4560 @mhw docs: Fix PostgreSQL typo in features.md

#4562 @gabe-lyons feat(lineage): show fully qualified dataset name on expansion

#4561 @anshbansal fix: dependencies for usage sources

#3782 @CorentinDuhamel feat(ingest): indent sql queries for usage sources

#4551 @pedro93 fix(rollback) Removes status & key aspects from affected aspects count during rollback

#4538 @dexter-mh-lee fix(policy): Remove all from the resource type choices

#4544 @anshbansal fix(ingest): snowflake-usage - log warning instead of error out

#4542 @mattmatravers build(ui): allow custom nodeDistBaseUrl

#4545 @mayurinehate fix(kafka-connect): add platform for default case in jdbc connector, update tests for platform instance map

#4547 @anshbansal chore: update pull request template

#4537 @RyanHolstien fix(dataPlatformInstance): add data platform instance to entity registry

#4375 @mayurinehate fix(kafka-connect): fix lineage for postgres-like 3-level hierarchy d…

#4492 @RyanHolstien fix(cli): delete - handle case insensitive entity types

#4130 @sgomezvillamor feat(ingest): glue - adds platform instance capability

#4456 @mohdsiddique feat(stateful dbt): add stateful ingestion capability in dbt source

#4482 @pedro93 feat(platform): adds side-effect report for rollbacks

#4275 @maggiehays docs: Ingestion Source Docs Template

#4470 @mayurinehate feat(tableau): emit lineage edge from embedded datasource to upstream…

#4535 @pppsunil feat(ingestion): Support pluggable Schema Registry for Kafka Source

#4369 @eburairu feat(ui): Add new loading pattern logo

#4493 @leifker feat(integration): protobuf - additional annotations and features

#4435 @zhoxie-cisco perf(docker): datahub-gms - add jetty configuration xml

#4532 @anshbansal doc: add example of profiling in default example

#4533 @anshbansal doc: clarify CLI releases

#4528 @daha fix(doc): Change to forward slash-separated strings, as in the example

#4315 @daha fix(docs): Minor fixes to pip install commands

#4521 @anshbansal doc: update docker docs for mentioning Python CLI

#4523 @kevinhu fix(ingest): mssql - support database_alias

#4501 @arunvasudevan feat(ingest): kafka-connect - support mapping for multiple DB instances

#4477 @jjoyce0510 feat(metadata service): Introducing Platform Events

#4526 @jjoyce0510 Adding has container

#4494 @cuong-pham fix(ingest): make tableau ingestion more resilient to error

#4519 @anshbansal doc(ingestion): add examples of running in docker and Kubernetes

#4485 @andres-lowrie docs(metadata-ingestion): callout props in para

#4511 @RyanHolstien Oss/urn validation

#4525 @dexter-mh-lee feat(policy): Add tooltip and view button

#4507 @mayurinehate feat(assertion): update python example, assertion entity doc

#4513 @kevinhu feat(ingestion): detect and disable telemetry in CI

#4516 @dexter-mh-lee feat(policy): Add domain based and view based policies

#4520 @rslanka Fix: Snowflake Table to View lineage

#4510 @anshbansal feat(ingest): Add config to improve user exp for initial ingestion and fix docs

#4517 @anshbansal feat(ingest): option for number of workunits in preview

#4503 @treff7es feat(ingest): athena - set Athena location as upstream

#4490 @MugdhaHardikar-GSLab feat(s3): add s3 source

#4468 @tc350981 feat(notebook): graphqul related logic change for notebook

#4401 @anshbansal doc: update instructions for updating DataHub on quickstart

#4500 @ShubhamThakre SecretBuilderModal -> name field validation updated

#4505 @anshbansal docs: add example of database and schema allow/deny patterns

#4504 @anshbansal fix(snowflake): allow/deny patterns

#4496 @shirshanka feat(ingest): dbt,looker,sql_common,kafka - moving sources to produce display names and subtypes more consistently

#4480 @anshbansal feat(snowflake): stop querying for usage data when no mix/max dates

#4483 @anshbansal fix(snowflake-usage): do not ingest for stage as a dataaset

#4497 @shirshanka moving to dockerhub for actions container

#4475 @darapuk fix: Update GroupProfile to read from properties over deprecated info aspect

#4489 @anshbansal fix(ingestion): pin Jinja2 to version < 3.1.0

#4484 @anshbansal fix(ingestion): stop CLI build failures

#4467 @anshbansal doc: add caveats to snowflake doc

#4481 @anshbansal fix(snowflake): don't recommend accountadmin role for snowflake

#4479 @anshbansal fix: change log level to debug

#4476 @dexter-mh-lee Add recent searches filtering

#4469 @tc350981 feat(ingest): add python utility classes for NotebookUrn, CorpuserUrn and CorpGroupUrn

#4439 @ShubhamThakre Feature/modal-validation-and-UI-fixes-updates

#4398 @kevinhu feat(ingest): simplify event IDs for function invocations

#4474 @sgomezvillamor chore: acryl-data 0.6.12

#4473 @treff7es fix(redshift) Properly handling database alias in redshift usage and redshift lineage generation

#4471 @gabe-lyons enabling ml tabs

#4460 @eclaassen-pb fix: java dependency vulnerabilities

#4453 @treff7es feat(ingest) data-lake: Add s3 properties metadata when ingesting s3 files

#4464 @anshbansal fix: change for repository change

#4466 @anshbansal fix(snowflake-usage): add more error handling

#4445 @nandacamargo fix(ingest): add fix to tableau connector when table has None fields

#4457 @mayurinehate docs(hive): update recipe with example to specify kerberos auth

#4313 @pedro-iatzky fix(ingest): bigquery - fix ingestion of external tables

#4462 @gabe-lyons adding final transport options

#4223 @tc350981 feat(notebook): add data models for Notebook entity

#4450 @pedro93 feat(frontend) Adds multiple group claim support

#4442 @anshbansal feat(ingestion): snowflake, bigquery - enhancements to log and bugfix

#4333 @anshbansal doc: add guide for ui tabs

#4443 @jjoyce0510 Fixing privilege option display bug

#4451 @gabe-lyons fix tableau connector when it cannot connect to URI

#4237 @tc350981 (docs) add RFC file to introduce Notebook entity data model

#4446 @kevinneville fix: Replace old repository link with new link

#4447 @cuong-pham getting database directly from upstream tables incase there are multiple databases in upstreamDatabases

#4436 @treff7es Passing entity properly on deletion

#4433 @rslanka Fix bug in the SchemaField type computation for AVRO logical types.

0.8.2910 Mar 2022

This version is affected by an OIDC (SSO) related issue with the following stack trace:

v0.8.29

NOTICE

This version is affected by an OIDC (SSO) related issue with the following stack trace:

datahub-datahub-frontend-8d7f7cf6f-xvjwm datahub-frontend Caused by: java.security.InvalidKeyException: Invalid AES key length: 30 bytes
datahub-datahub-frontend-8d7f7cf6f-xvjwm datahub-frontend 	at com.sun.crypto.provider.AESCrypt.init(AESCrypt.java:87)

DataHub core team is working to address this. For now, we recommend staying on 0.8.28 if you are using OIDC actively!

Release Highlights

Fix for MAE & MCE consumer healthcheck Upgrade to Java 11 and Gradle 6

Full Commit Log

#4360 @maaaikoool Add cabify as adopter
#4365 @dexter-mh-lee fix(vulnerabilities): Fix vulnerabilities in datahub-frontend
#4361 @jjoyce0510 fix(ui): Supporting unknown data platform type
#4363 @rslanka feat(ingest): Add memory leak detection capability to the datahub cli command.
#4366 @RyanHolstien fix(metadata-jobs): fix root context for springboot
#4340 @leifker feat(build): upgrade to gradle 6 for toolchain to support java 11
#4357 @anshbansal feat: change quickstart to use head tag for actions
#4356 @treff7es fix(ingest): bigquery - Fixing missing attribute error if credential was not set
#4319 @vcs9 feat(ingest): mysql - add database_alias functionality
#4352 @dexter-mh-lee fix(ci): fix model generation workflow
#4351 @jjoyce0510 fix(frontend): Fix common OIDC issues
#4111 @treff7es fix(ingest) bigquery-usage: Adding credential support for bigquery usage
#4343 @Ankit-Keshari-Vituity Fixed the Small Project issue
#4350 @MugdhaHardikar-GSLab fix(config-parsing): add support for variable expansion for in variables in between string
#4330 @anshbansal fix(hive): clean protocol for hive source
#4338 @maggiehays doc(platforms) adding PowerBI logo to docs website'
#4342 @anshbansal feat(quickstart): restart actions pod in case of failures
#4347 @mayurinehate fix(GE): fix dependencies for GE DataHubValidationAction, logic for s…
#4349 @gabe-lyons query for custom properties on containers
#4341 @anshbansal fix(doc): remove duplicate entry for permission
#9 @shirshanka fix(ci): fix datahub jar publish action
#8 @shirshanka feat(ci): fix jar publish action
#7 @czbernard fix(ci): fixing tag computation for docker image build
#6 @shirshanka feat(ci): adding dockerfile and action for datahub-airflow image
#5 @shirshanka fix(ci): pin python version to 3.9.9 for release action
#4 @dexter-mh-lee fix(ci): docker-ingestion - update acryl workflow
#3 @shirshanka fix(pypi): fixing package metadata to reflect source and changelog correctly
#2 @shirshanka fix(ci): check for tagged reference to kick off pypi push

Open source →

0.8.2723 Feb 2022Deprecated

feat(deprecation): Entity Deprecation Backend by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4073

Release Highlights

Notable UI-Based Features

The User Page has a new look! You can now quickly filter & search for entities owned by a User, update/edit the user profile, and see details of which Groups the User belongs to. See it in action here.
Search for Entities by Owner - Easily filter search results by User/Group Owner
Edit existing Glossary Terms - you can now edit/update Glossary Term descriptions via the UI. Future work will allow creating Terms from the UI as well - stay tuned!
Improved Metadata Analytics - keep tabs on your DataHub entities across Domains, Platforms, Glossary Terms, Environments, & more. Check out the new & improved Analytics tab!

Notable Metadata Model & Ingestion-Based Features

ClickHouse integration is now incubating! This is a 100% Community-led integration - huge shoutout to @ne1r0n & @havramar for pushing initial code & moving this work through!
Kafka Stateful Ingestion - shoutout to @claudio-benfatto for building this out!
Extract Airflow Task Description - big thanks to @guidoturtu for the contrib!
BigQuery: profile latest Partition/Shard - We know that Data Profiling can be computationally expensive for partitioned/sharded BQ instances. We now support profiling only the latest partition/shard to minimize processing load.

Notable Docs Updates

NEW! Tips for Searching within DataHub - Ever wondered how to make the most of Searching within DataHub? Check out this doc put together by @xiphl
Improvements to Metadata Model Docs - This is a huge win for the Community - we’re taking a big step toward providing auto-generated & curated docs related to the Metadata Model - take a look here.

What's Changed

feat(deprecation): Entity Deprecation Backend by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4073
Fixed auto complete pr coments by @Ankit-Keshari-Vituity in https://github.com/linkedin/datahub/pull/4072
fix(ingestion): enforce correct behaviour for commit policy by @claudio-benfatto in https://github.com/linkedin/datahub/pull/4092
fix(aggregate): Fix NPE in aggregate api by @dexter-mh-lee in https://github.com/linkedin/datahub/pull/4095
add Haibo corp by @wangqinghuan in https://github.com/linkedin/datahub/pull/4082
fix(ingestion): Add psutil dependency required for stateful ingestion reporting. by @rslanka in https://github.com/linkedin/datahub/pull/4099
docs(kafka): add example for using domains, change for clarity by @anshbansal in https://github.com/linkedin/datahub/pull/4100
feat(ui): Add display name & title to editable corp user properties. by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4097
fix(ingestion): Enhance BigQuery source logging. by @rslanka in https://github.com/linkedin/datahub/pull/4101
fix(glossary terms): fix add glossary term flow by @gabe-lyons in https://github.com/linkedin/datahub/pull/4106
(docs) Add Zynga & Tableau logos by @maggiehays in https://github.com/linkedin/datahub/pull/4109
fix(ingestion): Add sql lineage to redshift-usage plugin by @dexter-mh-lee in https://github.com/linkedin/datahub/pull/4103
feat(ui): Add svg datahub satellite loading logo by @eburairu in https://github.com/linkedin/datahub/pull/4067
fix(ingestion): resolve oracle issue with large view definitions by @hsheth2 in https://github.com/linkedin/datahub/pull/4027
fix(ingest): ignore Postgres information_schema tables by default by @kevinhu in https://github.com/linkedin/datahub/pull/4069
fix(ingest) - close event loops in Okta source and add additional debug logging by @aditya-radhakrishnan in https://github.com/linkedin/datahub/pull/4077
chore(ingest): remove unused groupby_unsorted utility by @hsheth2 in https://github.com/linkedin/datahub/pull/4011
fix(docs): fixing metadata model doc generation script and updating png by @swaroopjagadish in https://github.com/linkedin/datahub/pull/4120
fix(ci): fix formatting in doc generation action yaml by @swaroopjagadish in https://github.com/linkedin/datahub/pull/4121
fix(ci): fix formatting for action yaml by @swaroopjagadish in https://github.com/linkedin/datahub/pull/4122
feat(Tags/Terms): Backend support for tag & term mutations by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4096
docs(backup): add doc for taking backup by @anshbansal in https://github.com/linkedin/datahub/pull/3917
fix(docs): make intro to metadata ingestion easier for beginners by @anshbansal in https://github.com/linkedin/datahub/pull/4039
fix(ingest) Athena: db filter was not applied by @treff7es in https://github.com/linkedin/datahub/pull/4127
fix(ui) - move book logo to right of glossary term by @aditya-radhakrishnan in https://github.com/linkedin/datahub/pull/4125
fix(docs) Fix doc on modelDocUpload by @daha in https://github.com/linkedin/datahub/pull/4112
fix(cypress): force clicks on tag mutation test by @gabe-lyons in https://github.com/linkedin/datahub/pull/4102
feat(ingest) Athena: Getting table properties for Athena datasets by @treff7es in https://github.com/linkedin/datahub/pull/4123
fix(logging): Fix Restli Logging Filter to print full stack trace on error by @dexter-mh-lee in https://github.com/linkedin/datahub/pull/4136
docs : markdown fixes for db retention table by @satyamkrishna in https://github.com/linkedin/datahub/pull/4133
docs : markdown fixes for db retention table by @satyamkrishna in https://github.com/linkedin/datahub/pull/4148
feat(ingestion): Kafka stateful ingestion by @claudio-benfatto in https://github.com/linkedin/datahub/pull/4028
fix(docs): update graphql docs to reference new graphql file by @gabe-lyons in https://github.com/linkedin/datahub/pull/4139
Feature/oss/update to v2 endpoints by @RyanHolstien in https://github.com/linkedin/datahub/pull/4128
fix(cli): add timeout for telemetry calls by @anshbansal in https://github.com/linkedin/datahub/pull/4135
chore(cli): update default cli version pinned in the UI based ingestion by @anshbansal in https://github.com/linkedin/datahub/pull/4150
fix(docs): fix example of delta lake by @anshbansal in https://github.com/linkedin/datahub/pull/4149
fix(ui): Fix cutoff profiling axis labels by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4154
feat(ingest): Glue - Support for domains and containers by @treff7es in https://github.com/linkedin/datahub/pull/4110
feat(ui): Host platform images on datahub-web-react by @ngamanda in https://github.com/linkedin/datahub/pull/4118
bug(seedData): adds a key to the root user seed data and fixes corner case check for missing key aspects by @RyanHolstien in https://github.com/linkedin/datahub/pull/4162
UI Fix: Modal close on Enter press, autofocus on modal, added split panel, alignment of button by @Ankit-Keshari-Vituity in https://github.com/linkedin/datahub/pull/4155
feat(ui): Edit glossary term descriptions via UI by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4156
Update querying-entities.md -> Documentation Error by @buggythepirate in https://github.com/linkedin/datahub/pull/4157
refactor(metadata-io/test): common ElasticsearchContainer and ability to override from environment. by @stephenp-gr in https://github.com/linkedin/datahub/pull/4152
feat(ingestion): Add support for snowflake view lineage. by @rslanka in https://github.com/linkedin/datahub/pull/4163
Update the doc to including options to include Views by @cuong-pham in https://github.com/linkedin/datahub/pull/4164
fix(ingest): Use lower-case dataset names in the dataset urns for all SQL-styled datasets. by @rslanka in https://github.com/linkedin/datahub/pull/4140
chore(ingestion): upgrade mypy by @hsheth2 in https://github.com/linkedin/datahub/pull/4141
ci(ingestion): fix airflow 1 deps for tox by @hsheth2 in https://github.com/linkedin/datahub/pull/4083
fix(ingest) Glue: Removing sqlalchemy dependency from glue by @treff7es in https://github.com/linkedin/datahub/pull/4168
fix(ingest) Athena: Generating propert containers for Athena by @treff7es in https://github.com/linkedin/datahub/pull/4167
Feature/users and groups UI updated as per new design by @ShubhamThakre in https://github.com/linkedin/datahub/pull/4134
chore(docs): various cleanup for docs-website by @hsheth2 in https://github.com/linkedin/datahub/pull/4143
bugfix(logging): reduce log noise from authentication chain by @RyanHolstien in https://github.com/linkedin/datahub/pull/4173
bug(glossaryTermLabels): fix glossary term labels missing and add cypress test by @RyanHolstien in https://github.com/linkedin/datahub/pull/4171
fixes(ui): Misc UI fixes + Adding Owners to Search Filters by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4175
BugFixes/user-and-groups-minor-ui-fixes by @ShubhamThakre in https://github.com/linkedin/datahub/pull/4181
feat(groups): Adding editable group properties in the backend by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4166
fix(python build): Pinning markupsafe by @treff7es in https://github.com/linkedin/datahub/pull/4188
feat(analytics): Improve analytics page by adding more charts regarding metadata ingested by @dexter-mh-lee in https://github.com/linkedin/datahub/pull/4176
docs(model): auto-generated docs and hand-written docs for the metada… by @swaroopjagadish in https://github.com/linkedin/datahub/pull/4189
minor fixes(ui): Small UI display fixes by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4190
fix(ui): Return empty search response on invalid characters in search by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4193
refactor(spark-lineage): enhance logging and documentation by @MugdhaHardikar-GSLab in https://github.com/linkedin/datahub/pull/4113
fix(ui): Correctly display user photo on "list users" screen. by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4195
fix(ingest) Snowflake: Handle external S3 bucket lineage for "External Tables". by @rslanka in https://github.com/linkedin/datahub/pull/4192
fix(Ingestion): Elastic http/https host support by @abiwill in https://github.com/linkedin/datahub/pull/4191
Pinning down elasticsearch to less than 8.0.0 by @pppsunil in https://github.com/linkedin/datahub/pull/4182
test(airflow): fix airflow version parsing by @hsheth2 in https://github.com/linkedin/datahub/pull/4142
fix(delete): Fixing NPE on delete urns path by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4197
docs(website): Move company logos into tabs categorized by industry by @jeffmerrick in https://github.com/linkedin/datahub/pull/4174
fix(profile) Bigquery: Setting bigquery temp schema if it is set to fix limit and offset in profiling by @treff7es in https://github.com/linkedin/datahub/pull/4161
feat(ingest): record bucketed profiling runtimes by @kevinhu in https://github.com/linkedin/datahub/pull/4068
(docs) How to search better with search bar by @xiphl in https://github.com/linkedin/datahub/pull/4200
feat(ingest) Bigquery: Ignore temporary tables from lineage and connect edges directly by @treff7es in https://github.com/linkedin/datahub/pull/4160
feat(ingest): Not failing on table/view ingestion error by @treff7es in https://github.com/linkedin/datahub/pull/4185
fix(ingest): map additional Postgres types by @kevinhu in https://github.com/linkedin/datahub/pull/4179
feat(lineage): Add feature for lineage to capture airflow task description. by @guidoturtu in https://github.com/linkedin/datahub/pull/4147
add initial clickhouse support by @ne1r0n in https://github.com/linkedin/datahub/pull/4057
ci: update rename-namespace.sh to specify /bin/bash by @stephenp-gr in https://github.com/linkedin/datahub/pull/4153
fix(ingest): superset - adding missing greenlet dep by @swaroopjagadish in https://github.com/linkedin/datahub/pull/4203
fix(docs): fix config typo for stateful ingestion by @jieqiu0630 in https://github.com/linkedin/datahub/pull/4202
fix(dbt): dont product key aspects if the entity has no other aspects by @gabe-lyons in https://github.com/linkedin/datahub/pull/4217
feat(ingest): Add support for non-default schema registry subject name strategies to the Kafka source by @rslanka in https://github.com/linkedin/datahub/pull/4215
fix(ingest): Revert use lower-case dataset names in the dataset urns for all SQL-styled datasets. by @rslanka in https://github.com/linkedin/datahub/pull/4218
Add pagination to group ownerships by @mmmeeedddsss in https://github.com/linkedin/datahub/pull/4199
feat(Spark-smoke-test): add spark smoke test by @MugdhaHardikar-GSLab in https://github.com/linkedin/datahub/pull/4158
feat(ingest): add Python libs for Urns by @tc350981 in https://github.com/linkedin/datahub/pull/4172
feat(GraphQL API): Adding group ownership by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4219
fix(ui): Wrap homepage cards on long text by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4220
fix(docs): add mention of list-runs command in CLI page by @anshbansal in https://github.com/linkedin/datahub/pull/4210
fix(ingestion): enable compat with avro 1.11 by @hsheth2 in https://github.com/linkedin/datahub/pull/4205

New Contributors

@Ankit-Keshari-Vituity made their first contribution in https://github.com/linkedin/datahub/pull/4072
@wangqinghuan made their first contribution in https://github.com/linkedin/datahub/pull/4082
@daha made their first contribution in https://github.com/linkedin/datahub/pull/4112
@satyamkrishna made their first contribution in https://github.com/linkedin/datahub/pull/4133
@ngamanda made their first contribution in https://github.com/linkedin/datahub/pull/4118
@buggythepirate made their first contribution in https://github.com/linkedin/datahub/pull/4157
@stephenp-gr made their first contribution in https://github.com/linkedin/datahub/pull/4152
@cuong-pham made their first contribution in https://github.com/linkedin/datahub/pull/4164
@pppsunil made their first contribution in https://github.com/linkedin/datahub/pull/4182
@guidoturtu made their first contribution in https://github.com/linkedin/datahub/pull/4147
@ne1r0n made their first contribution in https://github.com/linkedin/datahub/pull/4057
@jieqiu0630 made their first contribution in https://github.com/linkedin/datahub/pull/4202
@mmmeeedddsss made their first contribution in https://github.com/linkedin/datahub/pull/4199
@tc350981 made their first contribution in https://github.com/linkedin/datahub/pull/4172

Full Changelog: https://github.com/linkedin/datahub/compare/v0.8.26...v0.8.27

Open source →

0.8.258 Feb 2022withdrawn: Rest emitter has a bug. Fix coming in 0.8.25.2Security

fix(vulnerability): Upgrade gms base image by @dexter-mh-lee in https://github.com/linkedin/datahub/pull/3962

Known Issues

Adding Glossary Terms to schema fields does not work with this version due to a bug. Upgrade to v0.8.26 for the fix.

Release Highlights

Buckle up, folks! v0.8.25 brings some very exciting (and highly-requested!) updates.

Notable UI-Based Features

UI-based Ingestion - as demoed in December Town Hall, we now support creating, configuring, scheduling, & executing batch metadata ingestion using the DataHub user interface. This makes getting metadata into DataHub easier by minimizing the overhead required to operate custom integration pipelines.
Data Domains - DataHub now supports grouping data assets into logical collections called Domains. Domains are curated, top-level folders or categories where related assets can be explicitly grouped. Read the guide here!
Data Containers are now supported! This is the physical grouping of entities, ex. a Schema is a container of 1 or more Datasets; a Dashboard is a container of 1 or more Charts.

Notable Metadata Model & Ingestion-Based Features

Data Quality test results are now supported in the DataHub metadata model. This is the first milestone toward surfacing Dataset & Column-level Data Quality results in the UI (read full scope of work here). Future releases will include a Great Expectations integration & UI support - we’re on track to complete this in Q1 as planned.
Avro files are now supported in the Data Lake File ingestion source
Ingest metadata from multiple instances of the same platform type. This has been a very common use case within the Community - you can now differentiate multiple instances of the same platform type! If you already have pre-existing entries, use the datahub migrate command to migrate them over to platform instances.
Ignore users from Top Users calculation
- feat(ingestion): Adding ability to ignore users from top users calculation by @treff7es in https://github.com/linkedin/datahub/pull/3735
BigQuery - Data Profiling on only the latest partition/shard
- feat(ingestion) bigquery: Profiling only the latest partition/shard on bigquery by @treff7es in https://github.com/linkedin/datahub/pull/3930

(feat)(Business Glossary) add tabular schema and new UI for business glossary by @saxo-lalrishav in https://github.com/linkedin/datahub/pull/3813

Notable Fixes

Fix to support View in Looker * feat(looker): Adding optional Looker external url base url config by @jjoyce0510 in https://github.com/linkedin/datahub/pull/3985
fix(graphql): support group display name in ownership by @thomasplarsson in https://github.com/linkedin/datahub/pull/3979
fix(profiling): Enabling profiling for low cardinality number columns by @treff7es in https://github.com/linkedin/datahub/pull/3990
fix(ingestion): match default username for Azure OIDC and Azure ingestion source by @iasoon in https://github.com/linkedin/datahub/pull/3926

DataHub Usage Guides

docs(domains): Adding a User Guide for Domains by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4038
docs(ingest): Adding UI ingestion guide by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4048

What's Changed

fix(vulnerability): Upgrade gms base image by @dexter-mh-lee in https://github.com/linkedin/datahub/pull/3962
logging(frontend): Improve OIDC debug logs by @jjoyce0510 in https://github.com/linkedin/datahub/pull/3967
docs(delete): add curl request example to delete entity by @anshbansal in https://github.com/linkedin/datahub/pull/3928
fix(ingestion): match default username for Azure OIDC and Azure ingestion source by @iasoon in https://github.com/linkedin/datahub/pull/3926
Feature/dynamic platform icons by @RyanHolstien in https://github.com/linkedin/datahub/pull/3968
refactor(ingestion): remove duplicate aspect type by @hsheth2 in https://github.com/linkedin/datahub/pull/3972
fix(example): fix typo by @anshbansal in https://github.com/linkedin/datahub/pull/3907
fix(ingestion): Restrict python to <=3.9.9 by @treff7es in https://github.com/linkedin/datahub/pull/3961
feat(build): remove requirement for git directory for builds by @swaroopjagadish in https://github.com/linkedin/datahub/pull/3977
fix(ingestion): tighten conditions for restli json transformation by @hsheth2 in https://github.com/linkedin/datahub/pull/3973
fix(ingestion): don't dump variables for config errors by @hsheth2 in https://github.com/linkedin/datahub/pull/3974
Bugfix/increase socket timeout by @RyanHolstien in https://github.com/linkedin/datahub/pull/3982
feat(ingest): support for Avro data lake files by @kevinhu in https://github.com/linkedin/datahub/pull/3913
fix(build): exclude old log4j core by @RickardCardell in https://github.com/linkedin/datahub/pull/3966
fix(quickstart): Pin Quickstart version to v0.8.23. by @jjoyce0510 in https://github.com/linkedin/datahub/pull/3983
feat(looker): Adding optional Looker external url base url config by @jjoyce0510 in https://github.com/linkedin/datahub/pull/3985
fix(graphql): support group display name in ownership by @thomasplarsson in https://github.com/linkedin/datahub/pull/3979
fix(quickstart): Assign correct mysql-setup container for M1 and remove "head" default version. by @jjoyce0510 in https://github.com/linkedin/datahub/pull/3987
feat(embedded search results): support custom endpoints in embedded search result by @gabe-lyons in https://github.com/linkedin/datahub/pull/3986
fix(docker): datahub-gms - build in native, copy to target by @swaroopjagadish in https://github.com/linkedin/datahub/pull/3992
fix(ci): moving defaults back to head now that docker builds are green by @swaroopjagadish in https://github.com/linkedin/datahub/pull/3993
feat(ui): UI-based ingestion (as featured in Dec Townhall) by @jjoyce0510 in https://github.com/linkedin/datahub/pull/3975
quickstart: Adding UI ingestion to quickstart YAML by @jjoyce0510 in https://github.com/linkedin/datahub/pull/3994
feat(domains): Adding backend for Asset Domains (p1) by @jjoyce0510 in https://github.com/linkedin/datahub/pull/3952
Bug: a bug fix to bigquery_to_datahub.yml file by @dipeshmaurya in https://github.com/linkedin/datahub/pull/3988
fix(ingest): check if feature data type is present by @maaaikoool in https://github.com/linkedin/datahub/pull/3932
feat(platform-instance): a simple client-only change to support platf… by @swaroopjagadish in https://github.com/linkedin/datahub/pull/3996
docs(metadata-model): Adding to Metadata model docs by @jjoyce0510 in https://github.com/linkedin/datahub/pull/3998
Add Stash Logo & new Source Icons by @maggiehays in https://github.com/linkedin/datahub/pull/4002
feat(domains): UI for Asset Domains (p2) by @jjoyce0510 in https://github.com/linkedin/datahub/pull/3995
docs: add missing back tick for metadata-ingestion/README.md by @nickwu241 in https://github.com/linkedin/datahub/pull/4003
Bugfix/add missing classes by @RyanHolstien in https://github.com/linkedin/datahub/pull/4000
fix(superset): fix connection for redshift by @anshbansal in https://github.com/linkedin/datahub/pull/3944
fix(setup): fix setup for M1 by @anshbansal in https://github.com/linkedin/datahub/pull/3958
docs:add Optum logo by @maggiehays in https://github.com/linkedin/datahub/pull/4005
Refining Metadata Model docs further by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4001
fix(docker): Alpine based multiplatform docker build for kafka-setup by @treff7es in https://github.com/linkedin/datahub/pull/3991
Bugfix/graph concurrency issue by @RyanHolstien in https://github.com/linkedin/datahub/pull/4007
feat(ingest): Add additional snowflake auth by @MikeSchlosser16 in https://github.com/linkedin/datahub/pull/4009
fix(ci): Reverting unnecessary domain test changes by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4013
fix(metrics): Add metrics for mcl hooks by @dexter-mh-lee in https://github.com/linkedin/datahub/pull/4008
feat(platform) - Update FabricType enum to represent more fabrics by @aditya-radhakrishnan in https://github.com/linkedin/datahub/pull/3997
feat(ingest): emit flags and stats for profiling telemetry by @kevinhu in https://github.com/linkedin/datahub/pull/3969
fix(formatting): fix linting lib version requirement by @anshbansal in https://github.com/linkedin/datahub/pull/3939
fix(docs): fix business glossary docs by @anshbansal in https://github.com/linkedin/datahub/pull/3916
fix(profiling): Enabling profiling for low cardinality number columns by @treff7es in https://github.com/linkedin/datahub/pull/3990
fix(docs): update gms link by @lhvubtqn in https://github.com/linkedin/datahub/pull/3927
fix(ingest): lint fix a few files by @swaroopjagadish in https://github.com/linkedin/datahub/pull/4016
fix(ingest): adding platform instance urn to data platform instance aspects by @swaroopjagadish in https://github.com/linkedin/datahub/pull/4015
feat(ingest): use trino python client for sqlalchemy, supports python… by @mayurinehate in https://github.com/linkedin/datahub/pull/3888
fix(spark-lineage): select mock server port dynamically for unit test by @MugdhaHardikar-GSLab in https://github.com/linkedin/datahub/pull/4018
(feat)(Business Glossary) add tabular schema and new UI for business glossary by @saxo-lalrishav in https://github.com/linkedin/datahub/pull/3813
Test/add concurrency issue smoke test by @RyanHolstien in https://github.com/linkedin/datahub/pull/4014
feat(glossary-terms): Index glossary term custom properties by @jjoyce0510 in https://github.com/linkedin/datahub/pull/3960
feat(ingestion): Adding ability to ignore users from top users calculation by @treff7es in https://github.com/linkedin/datahub/pull/3735
Docs/remote deploy and auto render by @RyanHolstien in https://github.com/linkedin/datahub/pull/4020
fix(ingest): snowflake - Run authentication validation if default value used by @treff7es in https://github.com/linkedin/datahub/pull/4024
feat(nifi): handle provenance api variation for older versions by @mayurinehate in https://github.com/linkedin/datahub/pull/4022
feat(ingestion) bigquery: Profiling only the latest partition/shard on bigquery by @treff7es in https://github.com/linkedin/datahub/pull/3930
fix(groups): Fix UI encoding of groups with spaces in urns by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4021
fix(text): fix confusing text by @anshbansal in https://github.com/linkedin/datahub/pull/4025
fix(clean): add missing cleanup by @anshbansal in https://github.com/linkedin/datahub/pull/4023
feat(containers): Backend for Asset Containers (as demo'd in townhall) by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4019
fix(docs): Adding Initiate login uri to okta docs (Okta OIDC) by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4030
fix: docker-compose now persists kafka broker data by @icy in https://github.com/linkedin/datahub/pull/4031
feat(ingestion): Support Kafka confluent external schema resolution by name or subject by @rslanka in https://github.com/linkedin/datahub/pull/4035
docs(domains): Adding a User Guide for Domains by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4038
feat(Stateful Ingestion-3/3): Client side changes for Monitoring/Reporting by @rslanka in https://github.com/linkedin/datahub/pull/3807
feat(containers): Adding Containers UI (as demo'd in Jan Townhall) by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4037
feat(users): adding user graphql mutation by @gabe-lyons in https://github.com/linkedin/datahub/pull/4033
feat(ingest): add tests for platform instance by @swaroopjagadish in https://github.com/linkedin/datahub/pull/4047
feat(model): Data quality model by @ksrinath in https://github.com/linkedin/datahub/pull/3787
Bugfix/prevent invalid urn by @RyanHolstien in https://github.com/linkedin/datahub/pull/4045
refactor(spark-lineage): remove dependency of spark from McpEmitter by @MugdhaHardikar-GSLab in https://github.com/linkedin/datahub/pull/4042
feat(analytics): add more analytics for entities by @anshbansal in https://github.com/linkedin/datahub/pull/4040
docs(ingest): Adding UI ingestion guide by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4048
fix(mae-consumer-docker): Fix condition for skipping elasticsearch check by @dexter-mh-lee in https://github.com/linkedin/datahub/pull/4052
feat(ci): pin tox requirements to speed up ci runs, remove airflow-1 … by @swaroopjagadish in https://github.com/linkedin/datahub/pull/4055
feat(container): Add domains aspect to container. by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4059
feat(profile) - bigquery: Fix for hitting limit with too many partitioned tables by @treff7es in https://github.com/linkedin/datahub/pull/4056
[Docs] Mark data lake metadata source as Beta by @pedro93 in https://github.com/linkedin/datahub/pull/4061
feat(ingest): log CLI invocations and completions by @kevinhu in https://github.com/linkedin/datahub/pull/4062
fix(ingest): Add aws dependencies for data lake by @kevinhu in https://github.com/linkedin/datahub/pull/4060
fix(ingest) - add aws_common as a snowflake_common dependency by @aditya-radhakrishnan in https://github.com/linkedin/datahub/pull/4054
feat(ui): Add svg datahub loading logo by @eburairu in https://github.com/linkedin/datahub/pull/4065
refactor(models): Refactoring new Assertion models by @jjoyce0510 in https://github.com/linkedin/datahub/pull/4064
feat(cli): add --force option to ingest rollback subcommand by @danilopeixoto in https://github.com/linkedin/datahub/pull/4032
fix(analytics): fix missing events from UI by @anshbansal in https://github.com/linkedin/datahub/pull/4026
Data domain containers ingestion by @treff7es in https://github.com/linkedin/datahub/pull/4051
docs(ingestion) glue: document required IAM permissions by @iasoon in https://github.com/linkedin/datahub/pull/3929
fix(profile):bigquery - Check for every table if it is partitioned to not hit table quota by @treff7es in https://github.com/linkedin/datahub/pull/4074

New Contributors

@dipeshmaurya made their first contribution in https://github.com/linkedin/datahub/pull/3988
@maaaikoool made their first contribution in https://github.com/linkedin/datahub/pull/3932
@icy made their first contribution in https://github.com/linkedin/datahub/pull/4031
@ksrinath made their first contribution in https://github.com/linkedin/datahub/pull/3787
@eburairu made their first contribution in https://github.com/linkedin/datahub/pull/4065
@danilopeixoto made their first contribution in https://github.com/linkedin/datahub/pull/4032

Full Changelog: https://github.com/linkedin/datahub/compare/v0.8.24...v0.8.25

Open source →

acryl-datahub

Release timeline

Releases

Release Highlights

User Experience

Ingestion Highlights

Full Commit Log

Full Commit Log

Release Highlights

Full Commit Log

Release Highlights

Notable UI-Based Features

Notable Metadata Model & Ingestion-Based Features

Notable Docs Updates

What's Changed

New Contributors

Known Issues

Release Highlights

Notable UI-Based Features

Notable Metadata Model & Ingestion-Based Features

Notable Fixes

DataHub Usage Guides

What's Changed

New Contributors