0.8.362 Jun 2022Deprecated

…users can now create, edit, move, delete, and deprecate Glossary Terms via the UI! With this new experience comes some new ways of indexing data in or…

V0.8.36

Highlights

User Experience

NEW – Manage Glossary Terms via the DataHub UI! Delivering on our Q2’22 Roadmap item, end users can now create, edit, move, delete, and deprecate Glossary Terms via the UI! With this new experience comes some new ways of indexing data in order to make viewing and traversing the different levels of your Glossary possible. Therefore, you will have to restore your indices in order for the new Glossary experience to work for users that already have existing Glossaries. If this is your first time using DataHub Glossaries, you're all set!

Ability to add multiple Owners, Tags, Terms

Developer Experience

The new Revokable Token API supports a new type of Access Token which can be revoked & queried, allowing admins to easily delete tokens for operational & security reasons. Read all about it in the Access Token Management Usage Guide.

Ingestion Updates

This release includes 3 new Metadata Sources:

Iceberg
Vertica
SAP HANA

📣 Massive shoutout to DataHub Community members @cccs-eric, @eburairu, and @buggythepirate for driving these contributions! 📣

These sources are currently marked as “Testing” - we encourage you to try them out & provide feedback in the DataHub #ingestion Slack channel!

We’ve rolled out the following ingestion-related improvements:

AWS Glue - data profiling is now supported
S3 ingestion speed-up
Various bug fixes

Full Commit Log

#5071 @dexter-mh-lee fix(docker): Fix mysql setup bug
#5066 @jjoyce0510 refactor(docs): Rename metadata modeling ingestion sidebar titles
#5036 @mmmeeedddsss fix(mysql-setup-job): add mysql default port override support
#5056 @nj7 fix: ES Rest Client Creation for non ssl authenticated connection
#5053 @ShubhamThakre fix(ui): ui bug fix for datasets sidebar stats section
#5061 @anshbansal feat(redash): add parallelism support for ingestion
#5017 @anshbansal feat(model): new chart types
#5047 @RyanHolstien fix(datahub-upgrade): exclude unnecessary configuration from standalone applications
#5052 @shirshanka feat(ci): datahub-client - add workflow, fix build
#5054 @jjoyce0510 docs(actions): Adding DataHub Actions to docs website
#5031 @piyushn-stripe feat(frontend): Allow overriding frontend with a custom akka http server
#5050 @dexter-mh-lee Remove exception on ingest policies
#5043 @Masterchen09 fix(docs): hana - rename SAP HANA source and data platform
#5051 @shirshanka fix(ingest): fix build breakage due to traitlets 5.2.2 bug
#5045 @anshbansal fix(redash): fix bug with names, add option for page size, debugging info
#5022 @jjoyce0510 fix(restore): Add RESTATE ChangeType to MCL / MCP to permit restore indices
#5041 @anshbansal doc(bigquery): fix missing permissions
#5030 @endeesa fix(doc) - Specify docker-compose version to avoid compatibility issues
#4879 @BoyuanZhangDE feat(ingest): glue - enable profiling
#5035 @treff7es fix(profiling): bigquery - Fix for Bigquery temp table creation on GE >= 0.15.3
#5040 @shirshanka fix(build): m1 build fails to install hdb-cli
#5026 @chriscollins3456 feat(glossary) Business Glossary updates
#4940 @MugdhaHardikar-GSLab fix(spark-lineage): remove need for sparksession.stop call
#5023 @rslanka fix(ingest): common - fix nullability determination for the AVRO fixed type.
#5012 @anshbansal fix(cli): don't use env for container, add example
#5021 @maggiehays docs(townhall): update townhall rsvp link and add may townhall detail
#5038 @shirshanka fix(build): docgen should fail if plugin is not loadable
#5033 @RyanHolstien fix(timelineAPI): fix issue with semantic versioning
#5034 @RyanHolstien fix(telemetry): exclude configuration from standalone apps
#5029 @RyanHolstien feat: telemetry improvements
#5028 @gabe-lyons dont set platform instances for sources
#5027 @anshbansal fix(parsing): incorrect parsing for commas
#4938 @Ankit-Keshari-Vituity refactor(ui): UI Integration to add multiple tags, terms and owners
#5025 @anshbansal fix(parsing): improve sql parsing, some debugging redash
#5024 @rslanka fix(ingestion): Remove hana from base_dev_requirements to unblock m1 users
#5014 @anshbansal fix(bigquery): reduce number of calls for details of partitioning
#5016 @ShubhamThakre fix(ui): arrow click position update
#5019 @rslanka fix(build): fix for hana build failure for aarch64.
#5020 @jjoyce0510 feat(Tests): Make DataHub Tests Feature configurable via env variable
#5005 @hsheth2 test(ingestion): change class names to avoid unittest warnings
#5006 @hsheth2 fix(ingestion): use raw strings for regexes
#5010 @rslanka feat(ingestion): Add Iceberg source
#5001 @PatrickfBraz fix(bigquery-usage): fix audit metadata query template
#4997 @anshbansal fix(redash): improve logging for debugging, add validation for dataset urn, some refactoring
#4376 @buggythepirate feat(ingest): Added new ingestion source SAP HANA
#5011 @rslanka Fix pulsar source docs.
#4555 @eburairu feat(ingest): Add Source from Vertica
#5008 @anshbansal fix(dbt): missing aws dependency
#5007 @anshbansal fix(bigquery): restrict protobuf version
#5004 @pedro93 fix(gms): Fix incorrect StatefulTokenService init
#5002 @ShubhamThakre fix(ui): ui bug fix - fixing search card vertical margin
#4994 @anshbansal doc(delete): add example for dataflow and datajob
#4988 @jjoyce0510 feat(DataHub Operations): Adding GraphQL mutation for reporting Dataset operations
#4998 @shirshanka fix(cli): timeline - adjust for timeline API changes on server
#5000 @pedro93 fix(docs): Fixes token docs
#4989 @jjoyce0510 feat(Tests): Metadata Tests Models + APIs + UI (Part 1)
#4995 @treff7es fix(airflow): Fix for Airflow 1 support
#4993 @shirshanka chore(deps): upgrade gson version
#4935 @BoyuanZhangDE feat(dbt): enable dbt read artifacts from s3
#4833 @treff7es feat(airflow): Airflow lineage ingestion plugin
#4931 @mayurinehate fix(ingest): tableau - fix chart custom properties None key error, update docs
#4943 @mayurinehate feat(model): add created, lastModified auditstamps to SchemaField
#4991 @anshbansal refactor(redash): emit charts first and try with id based dashboard API first
#4942 @mohdsiddique metabase chart are missing from dashboard
#4992 @anshbansal doc(ingest): update golden file command
#4927 @treff7es feat(ingest): s3 - speeding up ingestion with sampling
#4979 @pedro93 fix(smoke-tests) Increases sleep timeout in rollback test to prevent flakiness
#4964 @dexter-mh-lee feat(run): Create a describe run endpoint for fetching aspects created by the ingestion run
#4169 @claudio-benfatto feat(ingestion): optionally disable some kafka schema warnings
#4972 @mayurinehate feat(great-expectations): allow DATAHUB_DEBUG env var to enable debug logs in GE Action
#4957 @justinas-marozas refactor(metadata-io): introduce a storage-independent in-memory entity aspect model
#4982 @jjoyce0510 feat(authorization): Adding AuthorizerContext + ResourceSpecResolver to context
#4984 @anshbansal doc(ingestion): default boolean fix, broken bigquery docgen
#4970 @pedro93 feat(graphql) Add new Revokable Token API
#4987 @anshbansal fix(ingest): remove new schema field usage
#4985 @anshbansal fix(redash): use dashboard id if slug does not work
#4986 @pedro93 chore(deps): upgrade datastax libs version
#4981 @RyanHolstien fix(metadata-service): telemetry - fix hardcoded aspect name, suppress errors when producing MAE
#4983 @shirshanka fix(ingest): mode - dashboards without creator info fails to process
#4975 @chriscollins3456 fix(UI) Fix multiple UI usability issues
#4977 @maggiehays docs(townhall): update invite links and townhall history
#4980 @MugdhaHardikar-GSLab feat(spark-lineage): support for persist API
#4974 @anshbansal feat(bigquery): add partition key tag
#4967 @anshbansal fix(bigquery): add rate limiting for api calls made
#4971 @shirshanka fix(cli): graph - get_aspect_v2 method fails to deserialize aspects correctly
#4958 @anshbansal doc(ingest): mysql - describe required grants
#4969 @RyanHolstien doc(telemetry): fix telemetry doc
#4878 @MugdhaHardikar-GSLab fix(datahub-client): support utf8 encoding
#4961 @anshbansal feat(bigquery): reduce logging
#4909 @ShubhamThakre fix(ui): policy outside modal click issue update
#4968 @jeffmerrick docs(website): Remove banner and nav item for metadata day 2022
#4965 @mmmeeedddsss docs(datahub-kafka-sink): add topic_routes config to doc of datahub-kafka-sink
#4966 @liyuhui666 fix(data platforms): Update data_platforms.json
#4922 @mayurinehate feat(cli): raise error if get entity api fails
#4963 @Masterchen09 fix(ui): do not show copy URN buttons when Clipboard API is not available
#4962 @RyanHolstien feat(release): update CLI version
#4960 @RyanHolstien feat: updates for 0.8.35
#4945 @treff7es Revert "feat(spark-lineage): add support for iceberg and cache based plans (#4882)"
#4954 @dexter-mh-lee fix(ci): remove scheduled artifact deletion run to avoid api rate limiting
#4932 @anshbansal fix(bigquery): add dataset_id for bigquery
#4952 @RyanHolstien fix(metadata-service): timeline - ignore platform and schema changes
#4953 @dexter-mh-lee fix(ci): docker - remove multiplatform builds for unsupported images
#4950 @dexter-mh-lee fix(ci): add artifact cleaner, make docker publish sections consistent
#4947 @RyanHolstien fix(workflow): fix mysql credentials
#4951 @aditya-radhakrishnan fix(frontend): Update run-local-frontend to reflect the new Play changes
#4936 @gabe-lyons feat(transformers): add transformers to provide tags & terms to schema fields based on regex patterns
#4948 @dexter-mh-lee Fix docker unified
#4944 @gabe-lyons make graphql OperationType enum match up w/ pdl

Open source →

0.8.3518 May 2022Security

Reduced vulnerability counts in project Various bug fixes New streamlined docker workflow

Notice: Deploying this release will result in an incorrectly named aspect entry existing in the database. The impact is that some upgrade jobs may fail to perform full scans of the database. This will be fixed by upgrading to > v0.8.38 OR by pulling the latest DataHub Upgrade docker image and executing the following upgrade: ./datahub-upgrade.sh -u RemoveUnknownAspects

v0.8.35

Highlights

Reduced vulnerability counts in project Various bug fixes New streamlined docker workflow

Full Commit Log

#4937 @RyanHolstien fix(env): provide default for unset telemetry variable
#4926 @gabe-lyons feat(dbt): enable data platform instance on dbt
#4933 @anshbansal fix(lint): lint failure due to mypy upgrade
#4925 @RyanHolstien feat(telemetry): add server side telemetry
#4917 @jjoyce0510 feat(graphql): Adding resolvers for adding multiple tags, terms, and owners
#4924 @chen4119 fix(kafka-setup): Check if keystore/truststore location env variables are set
#4919 @jjoyce0510 feat(ui): Adding Search Bar to all List Views (groups, users, domains, policies, ingestion)
#4923 @chen4119 fix(kafka-setup): Add ssl.keystore.type and ssl.truststore.type
#4882 @maggie-zhu feat(spark-lineage): add support for iceberg and cache based plans
#4918 @RyanHolstien fix(idea): change location of coercer to make intellij not complain about classes
#4916 @chriscollins3456 fix(ui) Fix some spacing issues on the search card
#4914 @anshbansal docs(ingest): remove incorrectly annotated lineage capability
#4912 @mayurinehate docs(transformer): update custom transform example to add missing super init
#4903 @jjoyce0510 refactor(actions): Migrate to use new datahub-actions container
#4869 @jjoyce0510 refactor(API): Add "Filter" support for Assertion Run Events, Dataset Profiles, Dataset Operations
#4860 @anshbansal fix(doc): update doc url to generated docs
#4910 @chriscollins3456 feat(containers) Get and display all parent containers in header and search
#4791 @pedro93 feat(gms): Add support for deleting reference pointers when deleting by urn
#4911 @RyanHolstien docs(frontend): update build command for partial build
#4839 @BoyuanZhangDE feat(ingestion): For all usage connectors, allow exclusion of top_n_queries from ingestion via a config param.
#4908 @jeffmerrick fix(docs): Metadata day 2022: Fix year
#4859 @anshbansal doc(biqquery): add caveat for materialized view
#4906 @jeffmerrick docs(website): add banner and nav item for metadata day 2022
#4905 @anshbansal fix(build): Fix breaking changes from GE 0.15.3
#4884 @shirshanka fix(deps): reduce frontend dependency
#4902 @anshbansal doc(ingestion): add note for UI ingestion & custom sources
#4901 @anshbansal revert(bigquery-usage): dataset allow filter impl
#4824 @gabe-lyons fix(usage): pull usage from environment source rather than args
#4899 @SagarTiwari24 fix(docs): Update developing.md to mention directory context
#4892 @gabe-lyons fix(ui): fix side panel resize css
#4890 @justinas-marozas fix(mxe-consumer): exclude CassandraAutoConfiguration from consumer boot
#4853 @sebkim fix(ingestion): ElasticSearch when no properties from elastic_mappings, gracefully continue
#4865 @dependabot chore(deps): bump axios from 0.21.1 to 0.21.4 in /datahub-web-react
#4898 @treff7es fix(ingestion): bigquery-usage: Fix biquery usage table deny pattern template
#4893 @shirshanka fix(ci): remove logging statement
#4891 @RyanHolstien chore(deps): play - upgrade for CVEs
#4889 @shirshanka fix(ci): clean up docker workflow for multi-tags
#4875 @shirshanka fix(ingest): lookml - add view definitions for all views
#4887 @shirshanka fix(ci): docker - either load or push, don't do both
#4885 @shirshanka fix(ci): remove buildx and qemu for non multi-platform images
#4862 @anshbansal fix(sql-parsing): improve error handling
#4883 @shirshanka fix(ci): remove multiplatform builds from containers that don't support it
#4881 @shirshanka feat(ci): docker actions simplify, add vulnerability scanner, simplify smoke-tests
#4867 @chriscollins3456 feat(dataPlatformInstance) - Resolve and display dataPlatformInstance on entities
#4880 @shirshanka fix(docs): ingest - sort modules, fix small typos
#4866 @ShubhamThakre fix(ui): search filter entity ui update
#4855 @treff7es fix(ingestion): dependencies - Downgrading typing-extension dependency to work with Airflow 2.0.2
#4600 @pedro93 Use ingest proposal to submit status updates
#4868 @RyanHolstien Revert "chore(deps): upgrade play to remove CVEs (#4864)"
#4857 @RyanHolstien chore(jetty): upgrade jetty to 9.4.46 for CVE
#4776 @tha23rd fix(bigquery-usage): dataset allow filter impl
#4864 @RyanHolstien chore(deps): upgrade play to remove CVEs
#4843 @cristiancalugaru ssl configuration support for elasticsearch source
#4861 @RyanHolstien Revert "chore(deps): upgrade play dependencies to remove CVE vulnerabilities (#4820)"
#4846 @dependabot chore(deps): bump async from 2.6.3 to 2.6.4 in /docs-website
#4847 @dependabot chore(deps): bump minimist from 1.2.5 to 1.2.6 in /docs-website
#4820 @RyanHolstien chore(deps): upgrade play dependencies to remove CVE vulnerabilities
#4842 @rslanka fix(ingestion): Allow profiling of only those tables that are allowed by the table_pattern.
#4844 @RyanHolstien Revert "fix(jetty): upgrade jetty dependency for CVE (#4838)"
#4838 @RyanHolstien fix(jetty): upgrade jetty dependency for CVE
#4840 @rslanka chore(deps): upgrade dependency io.netty:netty-all to address vulnerability
#4841 @RyanHolstien fix(policies): change order of operations for policies bootstrap step to update index after database
#4837 @RyanHolstien chore(deps): move from velocity 1.7 to 2.3
#4821 @ShubhamThakre feat(ui): entity profile add copy url option update
#4817 @aditya-radhakrishnan docs(schema-history): add usage guide for schema history
#4835 @gabe-lyons hide soft deleted entities in lineage
#4836 @shirshanka refactor(metadata-service): remove redundant file
#4826 @jjoyce0510 chore(deps): pinning jackson dataformat cbor
#4777 @treff7es feat(ingest): s3 - add support for multiple pathspecs in one recipe
#4807 @eclaassen-pb chore(deps): upgrade spring and parquet dependencies
#4813 @pedro93 fix(docs): Adds access policy documentation
#4832 @mayurinehate feat(ingest): great-expectations - add more logs

Open source →

0.8.343 May 2022withdrawn: Incorrect commit hash used to create releaseSecurity

feat(vulnerability): Upgrade spring libraries to latest version by @dexter-mh-lee in https://github.com/datahub-project/datahub/pull/4698

Release Highlights

Developer Experience

DataHub Actions Framework is LIVE! The Actions Framework makes responding to real-time changes in your Metadata Graph easy, enabling you to seamlessly integrate DataHub into a broader events-based architecture. Check out the repo here
This release also introduces OpenAPI endpoints to post, get, and delete entities. Check out the usage guide here
Metadata Ingestion Source docs have a new look! We now have code-generated documentation to apply consistency in format and contents

User Experience

New! The Dataset Schema page now supports a “Blame View” to quickly understand how a field has evolved over semantic schema versions. You can find more info about how we compute versions here.

Ingestion Improvements

New! Now incubating the Apache Pulsar source
Update to Feast connector to support v0.18
Ongoing improvements to Snowflake external table support
Improvements to handling BigQuery audit log SQL queries
Miscellaneous Tableau fixes for lineage, browse path, non-embedded datasets

What's Changed

fix(cypress) - enable retries for failed tests to minimize flaking by @aditya-radhakrishnan in https://github.com/datahub-project/datahub/pull/4680
Deprecate an entity by @Ankit-Keshari-Vituity in https://github.com/datahub-project/datahub/pull/4633
fix(timeline): enhance schema field name change and removal support by @RyanHolstien in https://github.com/datahub-project/datahub/pull/4603
fix(cli): rest emitter should override config and env variables by @anshbansal in https://github.com/datahub-project/datahub/pull/4622
fix(docs): elasticsearch secret reference by @felixb in https://github.com/datahub-project/datahub/pull/4314
fix(mcl-processor): Remove unnecessary log.info by @dexter-mh-lee in https://github.com/datahub-project/datahub/pull/4686
fix(datahub-client): avoid parallel execution of metadat-io:test by @MugdhaHardikar-GSLab in https://github.com/datahub-project/datahub/pull/4685
docs(metadata-models-custom): add example script to show producing cu… by @shirshanka in https://github.com/datahub-project/datahub/pull/4681
fix(gms): Ensure Ordering by version when fetching next version by @arunvasudevan in https://github.com/datahub-project/datahub/pull/4696
fix(docker): Fix issue #4683 by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4697
feat(vulnerability): Upgrade spring libraries to latest version by @dexter-mh-lee in https://github.com/datahub-project/datahub/pull/4698
refactor(gms): EbeanAspectDao - make the orderBy clause explicitly ascending in getNextVersions by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4699
feat(gms): Entity change events v1 (Platform Event) by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4687
Redesign the login page by @Ankit-Keshari-Vituity in https://github.com/datahub-project/datahub/pull/4684
fix(snowflake): remove extra lineage edges in reports, change badly named config variable by @anshbansal in https://github.com/datahub-project/datahub/pull/4595
fix(bigquery): error due to not handling data properly by @anshbansal in https://github.com/datahub-project/datahub/pull/4702
fix(looker): Fix for Pydantic validation error for Looker TransportOptions on python 3.8 by @treff7es in https://github.com/datahub-project/datahub/pull/4705
fix(ingest) bigquery: Moving bigquery temporary credential deletion to atexit by @treff7es in https://github.com/datahub-project/datahub/pull/4701
fix(lineage): Fix lineage entity drawer height UI bug by @chriscollins3456 in https://github.com/datahub-project/datahub/pull/4707
feat(ingest) - update identity sources to add flags for masking sensitive work units by @aditya-radhakrishnan in https://github.com/datahub-project/datahub/pull/4711
fix(snowflake): deprecate config, update examples by @anshbansal in https://github.com/datahub-project/datahub/pull/4644
fix(glue): delete CatalogId parameter from get_jobs api call by @BoyuanZhangDE in https://github.com/datahub-project/datahub/pull/4646
fix(ui): Show deprecate button only for specific entity pages. by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4712
feat(ml): show custom properties for MLFeatureTable in UI by @maaaikoool in https://github.com/datahub-project/datahub/pull/4706
fix(glue): fix error for custom connector if ignore_unsupported_conne… by @mayurinehate in https://github.com/datahub-project/datahub/pull/4667
feat(ingest): add decimal128 custom type for mysql by @kevinhu in https://github.com/datahub-project/datahub/pull/4624
fix(policy): Use search to fetch all policies by @dexter-mh-lee in https://github.com/datahub-project/datahub/pull/4713
fix(transformers): add snapshot aspects from dataset into base_transf… by @shirshanka in https://github.com/datahub-project/datahub/pull/4719
Revert "fix(policy): Use search to fetch all policies" by @dexter-mh-lee in https://github.com/datahub-project/datahub/pull/4725
minor fix(metadata-ingestion): Add new schemas to python codegen by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4726
fix(ui): Display warning in UI when metadata service auth is disabled. by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4728
fix(timelineCli): fix naming for timeline cli by @RyanHolstien in https://github.com/datahub-project/datahub/pull/4729
fix(entity header): Fixes two issues in the EntityHeader - update UI and remove link by @chriscollins3456 in https://github.com/datahub-project/datahub/pull/4720
Revert "fix(timelineCli): fix naming for timeline cli (#4729)" by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4731
feat(cli): suppress stacktrace printing on configuration errors by @shirshanka in https://github.com/datahub-project/datahub/pull/4718
fix(cli): align default sink env variables across ingest and other cl… by @shirshanka in https://github.com/datahub-project/datahub/pull/4739
feat(ingest) dbt: Dbt query tag mapping and match template by @treff7es in https://github.com/datahub-project/datahub/pull/4744
fix(cli): telemetry - make config file processing more robust by @shirshanka in https://github.com/datahub-project/datahub/pull/4738
feat(react theming): stop homepage flicker for env-var based logos by @gabe-lyons in https://github.com/datahub-project/datahub/pull/4730
feat(Cassandra): add Cassandra implementation of EntityService by @xdl in https://github.com/datahub-project/datahub/pull/3286
fix(policies): Re-revert the policies fix + ingest documents directly to search by @dexter-mh-lee in https://github.com/datahub-project/datahub/pull/4733
feat(cli): Eagerly load datahub actions CLI commands by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4748
fix(ingest) bigquery: Fix BigQuery Datetime/Timestamp type column partition table profile bug by @sebkim in https://github.com/datahub-project/datahub/pull/4658
docs: add missing PR numbers by @anshbansal in https://github.com/datahub-project/datahub/pull/4742
fix(azure_ad): silently discard other Azure AD object types (#4693) by @cccs-eric in https://github.com/datahub-project/datahub/pull/4704
fix(datahub-frontend): OIDC discovery URL will not have NONE as auth_methods_supported by @chen4119 in https://github.com/datahub-project/datahub/pull/4710
fix(docs): fix links by @daha in https://github.com/datahub-project/datahub/pull/4703
feat(ingest): add Feast repository source by @danilopeixoto in https://github.com/datahub-project/datahub/pull/4094
feat(soft deletes): rephrasing soft delete banner by @gabe-lyons in https://github.com/datahub-project/datahub/pull/4753
feat(ebeans): Add metrics to track connection pool by @dexter-mh-lee in https://github.com/datahub-project/datahub/pull/4755
fix(AWS) When using aws_profile, grab temporary credentials from the session. by @Jiafi in https://github.com/datahub-project/datahub/pull/4751
feat(metadata-ingestion): Custom endpoint url and proxies in S3. by @pawel3275 in https://github.com/datahub-project/datahub/pull/4708
fix(tableau): miscellaneous tableau fixes for lineage, browse path, non-embedded datasets by @mayurinehate in https://github.com/datahub-project/datahub/pull/4724
doc: add warning for JDK by @anshbansal in https://github.com/datahub-project/datahub/pull/4761
fix(ui): fix expandedName for dataset by @mayurinehate in https://github.com/datahub-project/datahub/pull/4762
fix(ui): Users and Groups UI bug fixes by @ShubhamThakre in https://github.com/datahub-project/datahub/pull/4746
fix(azure_ad): make redirect and graph_url optional parameters and update docs by @aditya-radhakrishnan in https://github.com/datahub-project/datahub/pull/4754
docs(glue): clarify that table regex patterns should be fully-qualified by @aditya-radhakrishnan in https://github.com/datahub-project/datahub/pull/4747
fix(ml models): fix features tab by @gabe-lyons in https://github.com/datahub-project/datahub/pull/4769
fix(lint): lib upgrade caused by @anshbansal in https://github.com/datahub-project/datahub/pull/4773
fix(lineage) Filter dataset -> dataset lineage edges if data is transformed by @chriscollins3456 in https://github.com/datahub-project/datahub/pull/4732
fix(build): Fix breaking changes from GE 0.15.3 that are affecting our Python3.6 smoke_tests by @rslanka in https://github.com/datahub-project/datahub/pull/4779
fix(ingestion): Fixing how we eagerly import DataHub actions by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4784
fix(ingest): fwk - datahub_api should be initialized by datahub-rest … by @shirshanka in https://github.com/datahub-project/datahub/pull/4786
feat(ingestion) Ingest Tags from s3 bucket on an AWS Glue job and S3 Data Lake Ingest Job by @Jiafi in https://github.com/datahub-project/datahub/pull/4689
fix(snowflake): improve debug log for external tables by @anshbansal in https://github.com/datahub-project/datahub/pull/4772
feat(snowflake): add option to disable checking role grants by @anshbansal in https://github.com/datahub-project/datahub/pull/4760
fix(m1): tweak m1 preflight by @anshbansal in https://github.com/datahub-project/datahub/pull/4771
feat(ingestion): add Pulsar source by @vanmeete in https://github.com/datahub-project/datahub/pull/4721
fix(mae consumer): Fixes delete logic in MAE consumer by @pedro93 in https://github.com/datahub-project/datahub/pull/4790
feat(analytics): display glossary term percentage coverage by @aditya-radhakrishnan in https://github.com/datahub-project/datahub/pull/4782
refactor(gms): Removing unused source field by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4781
feat(versionedDataset): adds a versionStamp to timeline response & adds versionStamp param to dataset graphql by @RyanHolstien in https://github.com/datahub-project/datahub/pull/4727
fix(s3): improved handling for corner cases by @mayurinehate in https://github.com/datahub-project/datahub/pull/4774
fix(ingest): databricks - hive ingestion should not fail on table com… by @shirshanka in https://github.com/datahub-project/datahub/pull/4787
fix(ui ingest): Unschedule all sources on ingestion source refresh, fix delete not being enforced by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4792
feat(tracking) Configure whether mixpanel is enabled with env variable by @chriscollins3456 in https://github.com/datahub-project/datahub/pull/4768
feat(ingest): docs - overhaul source connector docs to make it code driven by @shirshanka in https://github.com/datahub-project/datahub/pull/4798
fix(docs): Fixing outdated control-center doc on policies.md by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4799
fix(ui): update default preview component with new ui design by @ShubhamThakre in https://github.com/datahub-project/datahub/pull/4783
feat(operation): display the reported time for last updated in the UI by @aditya-radhakrishnan in https://github.com/datahub-project/datahub/pull/4800
feat(blame) - add schema history blame UI by @aditya-radhakrishnan in https://github.com/datahub-project/datahub/pull/4793
fix(ingestion): Fix schema field type for avro logical types by @rslanka in https://github.com/datahub-project/datahub/pull/4801
Create sample_pii_glossary.yml by @mitchelllovessoftware123 in https://github.com/datahub-project/datahub/pull/4795
fix(ingestion): Fix presto_on_hive tests. by @rslanka in https://github.com/datahub-project/datahub/pull/4802
fix(bigquery): improve handling of extracted audit log sql queries by @vgaidass in https://github.com/datahub-project/datahub/pull/4735
fix(snowflake): get external tables when there is default namespace by @anshbansal in https://github.com/datahub-project/datahub/pull/4803
fix(snowflake): passing connect args should not cause failures by @anshbansal in https://github.com/datahub-project/datahub/pull/4764
fix(scrolling) Fixes scrolling and weird heights for embeddedListSearch across entities by @chriscollins3456 in https://github.com/datahub-project/datahub/pull/4805
fix(ui): update default preview card description text by @ShubhamThakre in https://github.com/datahub-project/datahub/pull/4796
fix(ui): preview card UI design update by @ShubhamThakre in https://github.com/datahub-project/datahub/pull/4808
fix(blame): make view blame prior to button work properly by @aditya-radhakrishnan in https://github.com/datahub-project/datahub/pull/4810
fix(docgen): fix failure count incrementing during doc generation by @shirshanka in https://github.com/datahub-project/datahub/pull/4806
fix(search) Fixes a UI issue so results and filters are always separated by @chriscollins3456 in https://github.com/datahub-project/datahub/pull/4811
feat(openapi): initial post,get, and delete endpoints for entities by @RyanHolstien in https://github.com/datahub-project/datahub/pull/4775
feat(protobuf) Adding deprecation support for datasets and fields by @leifker in https://github.com/datahub-project/datahub/pull/4634

New Contributors

@felixb made their first contribution in https://github.com/datahub-project/datahub/pull/4314
@chriscollins3456 made their first contribution in https://github.com/datahub-project/datahub/pull/4707
@sebkim made their first contribution in https://github.com/datahub-project/datahub/pull/4658
@chen4119 made their first contribution in https://github.com/datahub-project/datahub/pull/4710
@Jiafi made their first contribution in https://github.com/datahub-project/datahub/pull/4751
@pawel3275 made their first contribution in https://github.com/datahub-project/datahub/pull/4708
@vanmeete made their first contribution in https://github.com/datahub-project/datahub/pull/4721
@mitchelllovessoftware123 made their first contribution in https://github.com/datahub-project/datahub/pull/4795
@vgaidass made their first contribution in https://github.com/datahub-project/datahub/pull/4735

Full Changelog: https://github.com/datahub-project/datahub/compare/v0.8.33...v0.8.34

Open source →

0.8.3315 Apr 2022Breaking

doc: add page for handling deprecations, breaking changes etc. by @anshbansal in https://github.com/datahub-project/datahub/pull/4590

Release Highlights

User Experience

Refreshed the ML Entity page to match the feel of all other entity types; improved ML lineage functionality

Ingestion Improvements

Airflow Improvements - as demoed in March Town Hall
- Add support to capture Airflow execution runs from lineage backend
- Introduce new High level API for generating dataflow/job/dataprocessinstance
MS SQL ingestion now captures table & column descriptions
Trino platform support for Great Expectations
New Presto-on-Hive ingestion source
BigQuery ingestion now supports extraction of usage info from audit logs
Fix to Looker ingestion to extract Explore Views from join names
Fix to Tableau ingestion to avoid duplicating schema in URNs for upstream tables
Simplify & annotate Redshift Usage source

Full Commit Log

feat(gms): Expose kafka listener concurrency as a GMS setting by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4536
feat(ingest): add option for external Spark cluster by @kevinhu in https://github.com/datahub-project/datahub/pull/4571
fix(upgrade): Renaming kafka producer since it clashes with spring-internal by @dexter-mh-lee in https://github.com/datahub-project/datahub/pull/4573
feat(GraphQL): Add data platform query to GraphQL API by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4574
build(ui): Fix Windows UI lint by @mattmatravers in https://github.com/datahub-project/datahub/pull/4556
doc: make note prominent on quickstart by @anshbansal in https://github.com/datahub-project/datahub/pull/4558
fix(protobuf) minor bugfixes for protobuf by @leifker in https://github.com/datahub-project/datahub/pull/4553
feat(docs) Improves docs around developing datahub, removes deprecated docs on building metadata service by @pedro93 in https://github.com/datahub-project/datahub/pull/4552
chore: cleanup extra file by @anshbansal in https://github.com/datahub-project/datahub/pull/4541
feat(snowflake): reduce permissions provisioned by default by @anshbansal in https://github.com/datahub-project/datahub/pull/4543
fix(ingestion): Redshift usage refactoring - simplify, annotate, fix bugs by @rslanka in https://github.com/datahub-project/datahub/pull/4572
fix(graphql): Adding PRE FabricType to GraphQL by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4582
feat(search) - add DATETIME FieldType by @aditya-radhakrishnan in https://github.com/datahub-project/datahub/pull/4407
fix(tableau): fix for incorrect schema returned by tableau api for sn… by @mayurinehate in https://github.com/datahub-project/datahub/pull/4577
chore: update default cli for managed ingestion by @anshbansal in https://github.com/datahub-project/datahub/pull/4581
feat(okta) - add support for filtering/searching when ingesting Okta groups and users by @aditya-radhakrishnan in https://github.com/datahub-project/datahub/pull/4586
doc(snowflake): add example of table pattern by @anshbansal in https://github.com/datahub-project/datahub/pull/4580
fix(doc): try to fix broken link by @daha in https://github.com/datahub-project/datahub/pull/4593
fix(bigquery): incorrect lineage when views are present by @anshbansal in https://github.com/datahub-project/datahub/pull/4568
feat(metadata-service): Supporting a configurable Authorizer Chain by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4584
fix(search): Make sure home page and search pages are consistent by @dexter-mh-lee in https://github.com/datahub-project/datahub/pull/4588
fix(browse): Reduce browse aggregation size by @dexter-mh-lee in https://github.com/datahub-project/datahub/pull/4601
doc: add page for handling deprecations, breaking changes etc. by @anshbansal in https://github.com/datahub-project/datahub/pull/4590
docs(GraphQL): fix typo by @Falci in https://github.com/datahub-project/datahub/pull/4605
feat(search): Add SearchScore annotation to use fields for search ranking by @dexter-mh-lee in https://github.com/datahub-project/datahub/pull/4596
feat(ingestion): Redshift Usage Source - simplify OperationalStats workunit generation. by @rslanka in https://github.com/datahub-project/datahub/pull/4585
feat(tableau): add some logic to normalize table names in tableau by @gabe-lyons in https://github.com/datahub-project/datahub/pull/4609
fix: urlencode slash in urns too by @daha in https://github.com/datahub-project/datahub/pull/4527
fix(bigquery): fix lineage bug, improve docs, add dataset filter config by @anshbansal in https://github.com/datahub-project/datahub/pull/4607
fix(protobuf) fix test instabilitity by @leifker in https://github.com/datahub-project/datahub/pull/4612
fix(ui): Fix dashboard tags display by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4611
feat(ui): Adding GraphQL queries to fetch entity deprecation status by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4614
feat(ingest): enable connection string for all sqlalchemy datasources by @ms32035 in https://github.com/datahub-project/datahub/pull/4508
fix(docs): add grant statements for redshift-ingestion by @Abhiram98 in https://github.com/datahub-project/datahub/pull/4559
chore: fix lint and remove incorrect integration mark from unit tests by @anshbansal in https://github.com/datahub-project/datahub/pull/4621
feat: adding gradle, pip cache via gh cache, docker cache via dockerhub by @anshbansal in https://github.com/datahub-project/datahub/pull/4387
doc(scheduling): make it easier to find ui ingestion by @anshbansal in https://github.com/datahub-project/datahub/pull/4610
feat(glue): add CatalogId parameter for cross-account access by @BoyuanZhangDE in https://github.com/datahub-project/datahub/pull/4608
doc(cli): add env variables and options for ingest command by @anshbansal in https://github.com/datahub-project/datahub/pull/4598
fix(ingest): Restricting pytest docker version to <0.12 by @treff7es in https://github.com/datahub-project/datahub/pull/4639
fix(cypress) - add waits for cypress search test to remove flakiness by @aditya-radhakrishnan in https://github.com/datahub-project/datahub/pull/4640
Revert "feat: adding gradle, pip cache via gh cache, docker cache via dockerhub" by @dexter-mh-lee in https://github.com/datahub-project/datahub/pull/4637
feat(search): Only reindex if the mappings for an existing field changed by @dexter-mh-lee in https://github.com/datahub-project/datahub/pull/4629
feat: add presto-on-hive metadata ingestion source by @jchen0824 in https://github.com/datahub-project/datahub/pull/4625
feat(ingest): add trino platform for great expectations by @ms32035 in https://github.com/datahub-project/datahub/pull/4594
fix(kafka): Stop overriding kafka registry props with empty values by @jsotelo in https://github.com/datahub-project/datahub/pull/4604
[model]: Dataprocess instance entity to model datajob/jobflow runs by @treff7es in https://github.com/datahub-project/datahub/pull/4459
feat(ingest): add Urn python library for DataJob, DataFlow, Domain and Tag by @tc350981 in https://github.com/datahub-project/datahub/pull/4618
fix(ingestion): ensure source/sink reports are always logged by @anshbansal in https://github.com/datahub-project/datahub/pull/4592
fix(ingestion): extract explore views from join name in Looker by @dyanarose in https://github.com/datahub-project/datahub/pull/4627
feat(ingestion): Enable lower-casing of the name part of dataset urn if env variable is set. by @rslanka in https://github.com/datahub-project/datahub/pull/4649
feat: Enable the ingestion of bigquery audit logs to parse usage info… by @tha23rd in https://github.com/datahub-project/datahub/pull/4441
fix(ingest): Fix snowflake KEY_PAIR auth by @mkamalas in https://github.com/datahub-project/datahub/pull/4638
fix(home): Fix issue where some browse cards are missing by @dexter-mh-lee in https://github.com/datahub-project/datahub/pull/4652
fix(tableau): avoid duplicate schema in URNs for upstream tables by @maaaikoool in https://github.com/datahub-project/datahub/pull/4645
feat(ingest): capture MSSQL table+column descriptions by @kevinhu in https://github.com/datahub-project/datahub/pull/4579
feat(ml): bringing ml screens up to date w/ the modern ui layout & improving ml lineage by @gabe-lyons in https://github.com/datahub-project/datahub/pull/4651
(feat:airflow) Add support to capture airflow executions + high level dataflow/jobs api by @treff7es in https://github.com/datahub-project/datahub/pull/4615
fix(ingestion): add missing workunit ids by @anshbansal in https://github.com/datahub-project/datahub/pull/4657
fix(ingestion): Adding missing init.py by @anshbansal in https://github.com/datahub-project/datahub/pull/4659
fix(bigquery-usage): missing dependency by @anshbansal in https://github.com/datahub-project/datahub/pull/4661
feat(cypress) - add cypress dashboard view to CI by @aditya-radhakrishnan in https://github.com/datahub-project/datahub/pull/4654
feat(autocomplete): show fully qualified name in autocomplete by @gabe-lyons in https://github.com/datahub-project/datahub/pull/4663
feat(ingestion) dbt: Fixing issue with strip_user_ids_from_email and adding owner_naming_pattern by @arunvasudevan in https://github.com/datahub-project/datahub/pull/4587
fix(sqlparser): fix sqlparser breaking due to # sign by @anshbansal in https://github.com/datahub-project/datahub/pull/4662
fix(ingestion): validate datasource in Tableau connector, before creating its upstream by @nandacamargo in https://github.com/datahub-project/datahub/pull/4613
Added Relative Routing on the Users & Groups screen by @Ankit-Keshari-Vituity in https://github.com/datahub-project/datahub/pull/4664
fix(airflow): Not importing emitters directly to eliminate unneeded dependency by @treff7es in https://github.com/datahub-project/datahub/pull/4668
docs: remove ingestion source summary table by @maggiehays in https://github.com/datahub-project/datahub/pull/4670
feat(ml): some machine learning followups by @gabe-lyons in https://github.com/datahub-project/datahub/pull/4669
fix(search): Fix urn component settings by @dexter-mh-lee in https://github.com/datahub-project/datahub/pull/4672
fix(ingestion): update example recipes by @anshbansal in https://github.com/datahub-project/datahub/pull/4660
feat(theming): set custom logo without rebuilding by @gabe-lyons in https://github.com/datahub-project/datahub/pull/4674
feat(data-platform): Add platform entities for the connectors we support by @dexter-mh-lee in https://github.com/datahub-project/datahub/pull/4676
refactor(authorization): Add authorizedActor function to Authorizer interface by @dexter-mh-lee in https://github.com/datahub-project/datahub/pull/4678
docs(tags) - add tags usage guide by @aditya-radhakrishnan in https://github.com/datahub-project/datahub/pull/4677
fix(cli):Supress printing variables to logs during ingestion failure by @atulsaurav in https://github.com/datahub-project/datahub/pull/4566
fix(docs): Improving Add Users Doc by @jjoyce0510 in https://github.com/datahub-project/datahub/pull/4679
Fix/modal validations by @ShubhamThakre in https://github.com/datahub-project/datahub/pull/4673

New Contributors

@Falci made their first contribution in https://github.com/datahub-project/datahub/pull/4605
@ms32035 made their first contribution in https://github.com/datahub-project/datahub/pull/4508
@jchen0824 made their first contribution in https://github.com/datahub-project/datahub/pull/4625
@dyanarose made their first contribution in https://github.com/datahub-project/datahub/pull/4627
@mkamalas made their first contribution in https://github.com/datahub-project/datahub/pull/4638
@atulsaurav made their first contribution in https://github.com/datahub-project/datahub/pull/4566

Full Changelog: https://github.com/datahub-project/datahub/compare/v0.8.32...v0.8.33

Open source →

acryl-datahub

Release timeline

Releases

V0.8.36

Highlights

User Experience

Developer Experience

Ingestion Updates

Full Commit Log

Highlights

Full Commit Log

Release Highlights

Developer Experience

User Experience

Ingestion Improvements

What's Changed

New Contributors

Release Highlights

User Experience

Ingestion Improvements

Full Commit Log

New Contributors