NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #4746 most downloaded on PyPI
Official Python library for Unity Catalog AI support
Last release 5 months ago
24 Apr 2026
Release timing varies
gaps range from 4 weeks to 8 months
Most releases are documented
notes for 5 of 6 stable releases
Nothing withdrawn
no release was ever pulled
2 years old
8 releases · first in 2024
One column per month.
We are excited to announce Unity Catalog 0.4.0 release which includes exciting new features and many bug-fixes and improvements. A huge thank you to t
We are excited to announce Unity Catalog 0.4.0 release which includes exciting new features and many bug-fixes and improvements.
A huge thank you to the awesome community who made this release possible!
Artifacts in this release:
Unity Catalog server now provides full support for Storage Credentials for AWS — a first-class resource for managing cloud storage authentication. Storage credentials allow you to securely register and manage cloud provider credentials that Unity Catalog uses to govern access to the data. For example, UC will generate temporary credentials (time limited, and scoped down to a table/model/volume root directory) when an authorized user needs access to the data.
For AWS, Unity Catalog's supported credential type is the AWS IAM Role. Unity Catalog uses AWS Security Token Service ( STS) to assume this specified IAM role, which allows it to obtain temporary, scoped credentials for accessing AWS S3 cloud storage.
Managing credentials via UC CLI:
# Create a storage credential
bin/uc credential create --name my-s3-credential --aws-iam-role-arn arn:aws:iam::123456789012:role/my-data-role
# List all credentials
bin/uc credential list
# Update a credential
bin/uc credential update --name my-s3-credential --aws-iam-role-arn arn:aws:iam::123456789012:role/new-role
# Delete a credential (use --force to delete even if referenced by external locations)
bin/uc credential delete --name my-s3-credential --force
External locations are Unity Catalog objects that link a cloud storage path (for example, AWS S3) to a storage credential. They work together to enable secure, governed, and configurable access to cloud storage by vending temporary credentials instead of embedding secrets in table or volume definitions.
External locations can be used for creating any objects using cloud storage. It can be used as managed storage locations of catalogs and schemas which are used for creating MANAGED tables. It can also be used for creating EXTERNAL tables.
For each storage access, Unity Catalog automatically resolves the matching external location and vends short-lived, scoped credentials to the compute engine (for example, Spark). This centralizes credential management and ensures secure, temporary access to cloud storage.
Managing external locations via CLI:
# Create an external location
bin/uc external-location create \
--name my-s3-data \
--url s3://my-bucket/data \
--credential-name my-s3-credential
# List external locations
bin/uc external-location list
# Update an external location
bin/uc external-location update --name my-s3-data --url s3://my-bucket/new-data
# Delete an external location
bin/uc external-location delete --name my-s3-data
Unity Catalog server now supports setting a managed storage location when creating catalogs and schemas. When configured, managed Delta tables (experimental) and volumes created under that catalog or schema will automatically use storage under the parent’s managed storage location, making managed data placement consistent and easier to control.
Creating catalogs and schemas with Managed Storage Location via UC CLI:
# Create a catalog with managed storage location
bin/uc catalog create --name my_cat --storage_root s3://my-bucket/path
# Create a schema with managed storage location
bin/uc schema create --catalog some_other_cat --name my_schema --storage_root s3://my-bucket/path
In 0.3.1, automatic credential renewal for cloud storage (S3, Azure, GCS) required explicitly setting spark.sql.catalog.<catalog-name>.renewCredential.enabled=true. Starting with 0.4.0, credential renewal is enabled by default. Long-running Spark jobs will automatically renew storage credentials before they expire, preventing interruptions.
If you need to disable automatic renewal, you can explicitly set:
spark.sql.catalog.<catalog-name>.renewCredential.enabled=false
UC 0.4 now supports both Delta 4.1 with both Spark 4.0 and Spark 4.1. See Delta 4.1 release notes for more details on multiple Spark version support.
Before UC 0.4.0, CREATE TABLE AS SELECT (CTAS), REPLACE TABLE (RT), REPLACE TABLE AS SELECT (RTAS), CREATE OR REPLACE TABLE (CORT), and Dynamic Partition Overwrite (DPO) were non-atomic operations. If a failure occurred during data write or table replacement, the target table could be partially written, corrupted, or see existing data dropped.
Starting with UC 0.4.0 (used with Delta 4.1.0), Unity Catalog leverages Spark StagingTableCatalog API to enforce safer behavior:
A new unitycatalog-dspy package enables you to use Unity Catalog AI functions as tools within DSPy workflows. This allows LLM agents orchestrated by DSPy to call your registered UC functions as tools.
pip install unitycatalog-dspy
from unitycatalog.ai.dspy.toolkit import UCFunctionToolkit
import dspy
# Create a toolkit from your UC functions
toolkit = UCFunctionToolkit(
function_names=["catalog.schema.my_function"],
client=uc_client
)
# Use in a DSPy ReAct agent
agent = dspy.ReAct(
signature="question -> answer",
tools=toolkit.tools,
max_iters=5
)
Full Changelog: https://github.com/unitycatalog/unitycatalog/compare/v0.3.1...v0.4.0
The 0.4.0 release of Unity Catalog AI includes several improvements, including expanded Databricks client options and dependency/security improvements.
Warehouse ID Support in DatabricksFunctionClient: For workspaces where serverless compute is unavailable or restricted, DatabricksFunctionClient now accepts an optional warehouse_id parameter to dispatch function execution via the Statement Execution API. Serverless remains the default and recommended execution path.
Python 3.10+ Required: The minimum supported Python version has been raised to 3.10, as Python 3.9 has reached end-of-life.
unitycatalog-langchain by @bbqiu in #1120warehouse_id support to DatabricksFunctionClient by @smurching in #1428ai/core Python deps by @serena-ruan in #1457unitycatalog-ai dependency (remove redundant pydantic check, move pandas import) by @serena-ruan in #1519databricks-connect for Databricks integration by @aravind-segu in #1062unitycatalog-langchain and unitycatalog-openai package versions by @serena-ruan in #1515We are excited to announce Unity Catalog 0.4.0 release which includes exciting new features and many bug-fixes and improvements.
A huge thank you to the awesome community who made this release possible!
Artifacts in this release:
Unity Catalog server now provides full support for Storage Credentials for AWS — a first-class resource for managing cloud storage authentication. Storage credentials allow you to securely register and manage cloud provider credentials that Unity Catalog uses to govern access to the data. For example, UC will generate temporary credentials (time limited, and scoped down to a table/model/volume root directory) when an authorized user needs access to the data.
For AWS, Unity Catalog's supported credential type is the AWS IAM Role. Unity Catalog uses AWS Security Token Service ( STS) to assume this specified IAM role, which allows it to obtain temporary, scoped credentials for accessing AWS S3 cloud storage.
Managing credentials via UC CLI:
# Create a storage credential
bin/uc credential create --name my-s3-credential --aws-iam-role-arn arn:aws:iam::123456789012:role/my-data-role
# List all credentials
bin/uc credential list
# Update a credential
bin/uc credential update --name my-s3-credential --aws-iam-role-arn arn:aws:iam::123456789012:role/new-role
# Delete a credential (use --force to delete even if referenced by external locations)
bin/uc credential delete --name my-s3-credential --forceExternal locations are Unity Catalog objects that link a cloud storage path (for example, AWS S3) to a storage credential. They work together to enable secure, governed, and configurable access to cloud storage by vending temporary credentials instead of embedding secrets in table or volume definitions.
External locations can be used for creating any objects using cloud storage. It can be used as managed storage locations of catalogs and schemas which are used for creating MANAGED tables. It can also be used for creating EXTERNAL tables.
For each storage access, Unity Catalog automatically resolves the matching external location and vends short-lived, scoped credentials to the compute engine (for example, Spark). This centralizes credential management and ensures secure, temporary access to cloud storage.
Managing external locations via CLI:
# Create an external location
bin/uc external-location create \
--name my-s3-data \
--url s3://my-bucket/data \
--credential-name my-s3-credential
# List external locations
bin/uc external-location list
# Update an external location
bin/uc external-location update --name my-s3-data --url s3://my-bucket/new-data
# Delete an external location
bin/uc external-location delete --name my-s3-dataUnity Catalog server now supports setting a managed storage location when creating catalogs and schemas. When configured, managed Delta tables (experimental) and volumes created under that catalog or schema will automatically use storage under the parent’s managed storage location, making managed data placement consistent and easier to control.
Creating catalogs and schemas with Managed Storage Location via UC CLI:
# Create a catalog with managed storage location
bin/uc catalog create --name my_cat --storage_root s3://my-bucket/path
# Create a schema with managed storage location
bin/uc schema create --catalog some_other_cat --name my_schema --storage_root s3://my-bucket/pathIn 0.3.1, automatic credential renewal for cloud storage (S3, Azure, GCS) required explicitly setting spark.sql.catalog.<catalog-name>.renewCredential.enabled=true. Starting with 0.4.0, credential renewal is enabled by default. Long-running Spark jobs will automatically renew storage credentials before they expire, preventing interruptions.
If you need to disable automatic renewal, you can explicitly set:
spark.sql.catalog.<catalog-name>.renewCredential.enabled=false
UC 0.4 now supports both Delta 4.1 with both Spark 4.0 and Spark 4.1. See Delta 4.1 release notes for more details on multiple Spark version support.
Before UC 0.4.0, CREATE TABLE AS SELECT (CTAS), REPLACE TABLE (RT), REPLACE TABLE AS SELECT (RTAS), CREATE OR REPLACE TABLE (CORT), and Dynamic Partition Overwrite (DPO) were non-atomic operations. If a failure occurred during data write or table replacement, the target table could be partially written, corrupted, or see existing data dropped.
Starting with UC 0.4.0 (used with Delta 4.1.0), Unity Catalog leverages Spark StagingTableCatalog API to enforce safer behavior:
A new unitycatalog-dspy package enables you to use Unity Catalog AI functions as tools within DSPy workflows. This allows LLM agents orchestrated by DSPy to call your registered UC functions as tools.
pip install unitycatalog-dspyfrom unitycatalog.ai.dspy.toolkit import UCFunctionToolkit
import dspy
# Create a toolkit from your UC functions
toolkit = UCFunctionToolkit(
function_names=["catalog.schema.my_function"],
client=uc_client
)
# Use in a DSPy ReAct agent
agent = dspy.ReAct(
signature="question -> answer",
tools=toolkit.tools,
max_iters=5
)Note truncated.
Nothing published for this version
We are excited to announce Unity Catalog 0.3.1 release which includes exciting new features and many bug-fixes and improvements.
We are excited to announce Unity Catalog 0.3.1 release which includes exciting new features and many bug-fixes and improvements.
A huge thank you to the awesome community who made this release possible!
Artifacts in this release:
This release introduces a reusable unitycatalog-client Java module that provides an abstraction layer for building Unity Catalog API clients across different integrations:
Map<String, String> configs = Map.of("type", "static", "token", "my-unitycatalog-token");
ApiClient client = ApiClientBuilder.create()
.uri("http://localhost:8080")
.tokenProvider(TokenProvider.create(configs))
.retryPolicy(JitterDelayRetryPolicy.builder().maxAttempts(5).build())
.build();
Pluggable Authentication: Introduced TokenProvider interface with built-in support for user-provided tokens and OAuth configurations, and also with support for custom dynamically-loaded providers via specifying the class name (see TokenProvider documentation for more details).
Retry and Resiliency: Standardized retry handling with RetryPolicy interface for automatic transient failure recovery and set a jittered backoff retry policy as the default to help prevent excessive request pressure on the Unity Catalog server.
The unitycatalog-spark connector now supports automatic renewal of S3/Azure/GCS storage credentials. This feature ensures that long-running Spark jobs and Unity Catalog workloads can access cloud storage without interruption:
org.apache.hadoop:hadoop-aws:3.4.0 to spark external libraries and set the extra spark configurations (example):spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
spark.sql.catalog.<catalog-name>.renewCredential.enabled=true
org.apache.hadoop:hadoop-azure:3.3.6 to spark external libraries and set the extra spark configurations (example):spark.sql.catalog.<catalog-name>.renewCredential.enabled=true
com.google.cloud.bigdataoss:gcs-connector:3.0.2 to spark external libraries and set the extra spark configurations (example):spark.sql.catalog.<catalog-name>.renewCredential.enabled=true
The Unity Catalog Spark connector now supports OAuth authentication. When OAuth is enabled, the client automatically requests short-lived tokens using the configured OAuth endpoint, client ID, and client secret. It also detects token expiration and transparently renews tokens without interrupting user workloads. To enable it, set the following to your spark configurations:
spark.sql.catalog.<catalog-name>.auth.type=oauth
spark.sql.catalog.<catalog-name>.auth.oauth.uri=<oauth-uri>
spark.sql.catalog.<catalog-name>.auth.oauth.clientId=<oauth-client-id>
spark.sql.catalog.<catalog-name>.auth.oauth.clientSecret=<oauth-client-secret>
This release introduces experimental (work‑in‑progress) support for UC‑managed Delta tables, gated by the catalogManaged table feature (see Delta RFC for more details). A UC-managed Delta table has its storage location allocated and its commits coordinated by Unity Catalog server.
Staging Table API: A new API (/staging-tables) is introduced to allow creation of a staging table and allocation of a managed location that a managed table will be built on top of.
Delta Commit API: A new Delta commit API endpoint (/delta/preview/commits, both GET and POST) is introduced to coordinate Delta commits by UC server.
UCSingleCatalog Spark integration: Now it supports creation and data read & write of a managed Delta table from Delta-Spark with storage location allocated by UC server and commits coordinated by unity catalog server.
Note: This new feature is experimental and the APIs are subject to change in future release. We are releasing this for every one to start building integrations and connectors. To test this out, you can do the following:
Start the UC server with the server property server.managed-table.enabled=true.
Start Spark 4.0 with the upcoming Delta 4.0.1 release.
An example Spark SQL to create a managed table:
CREATE TABLE <table_name> ... USING DELTA TBLPROPERTIES('delta.feature.catalogManaged'='supported');
build/sbt generate by @yili-db in https://github.com/unitycatalog/unitycatalog/pull/1231Full Changelog: https://github.com/unitycatalog/unitycatalog/compare/v0.3.0...v0.3.1
We are excited to announce Unity Catalog 0.3.1 release which includes exciting new features and many bug-fixes and improvements.
A huge thank you to the awesome community who made this release possible!
Artifacts in this release:
This release introduces a reusable unitycatalog-client Java module that provides an abstraction layer for building Unity Catalog API clients across different integrations:
Map<String, String> configs = Map.of("type", "static", "token", "my-unitycatalog-token");
ApiClient client = ApiClientBuilder.create()
.uri("http://localhost:8080")
.tokenProvider(TokenProvider.create(configs))
.retryPolicy(JitterDelayRetryPolicy.builder().maxAttempts(5).build())
.build();Pluggable Authentication: Introduced TokenProvider interface with built-in support for user-provided tokens and OAuth configurations, and also with support for custom dynamically-loaded providers via specifying the class name (see TokenProvider documentation for more details).
Retry and Resiliency: Standardized retry handling with RetryPolicy interface for automatic transient failure recovery and set a jittered backoff retry policy as the default to help prevent excessive request pressure on the Unity Catalog server.
The unitycatalog-spark connector now supports automatic renewal of S3/Azure/GCS storage credentials. This feature ensures that long-running Spark jobs and Unity Catalog workloads can access cloud storage without interruption:
org.apache.hadoop:hadoop-aws:3.4.0 to spark external libraries and set the extra spark configurations (example):spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
spark.sql.catalog.<catalog-name>.renewCredential.enabled=true
org.apache.hadoop:hadoop-azure:3.3.6 to spark external libraries and set the extra spark configurations (example):spark.sql.catalog.<catalog-name>.renewCredential.enabled=true
com.google.cloud.bigdataoss:gcs-connector:3.0.2 to spark external libraries and set the extra spark configurations (example):spark.sql.catalog.<catalog-name>.renewCredential.enabled=true
The Unity Catalog Spark connector now supports OAuth authentication. When OAuth is enabled, the client automatically requests short-lived tokens using the configured OAuth endpoint, client ID, and client secret. It also detects token expiration and transparently renews tokens without interrupting user workloads.
To enable it, set the following to your spark configurations:
spark.sql.catalog.<catalog-name>.auth.type=oauth
spark.sql.catalog.<catalog-name>.auth.oauth.uri=<oauth-uri>
spark.sql.catalog.<catalog-name>.auth.oauth.clientId=<oauth-client-id>
spark.sql.catalog.<catalog-name>.auth.oauth.clientSecret=<oauth-client-secret>
This release introduces experimental (work‑in‑progress) support for UC‑managed Delta tables, gated by the catalogManaged table feature (see Delta RFC for more details). A UC-managed Delta table has its storage location allocated and its commits coordinated by Unity Catalog server.
Staging Table API: A new API (/staging-tables) is introduced to allow creation of a staging table and allocation of a managed location that a managed table will be built on top of.
Delta Commit API: A new Delta commit API endpoint (/delta/preview/commits, both GET and POST) is introduced to coordinate Delta commits by UC server.
UCSingleCatalog Spark integration: Now it supports creation and data read & write of a managed Delta table from Delta-Spark with storage location allocated by UC server and commits coordinated by unity catalog server.
Note: This new feature is experimental and the APIs are subject to change in future release. We are releasing this for every one to start building integrations and connectors. To test this out, you can do the following:
Start the UC server with the server property server.managed-table.enabled=true.
Start Spark 4.0 with the upcoming Delta 4.0.1 release.
An example Spark SQL to create a managed table:
CREATE TABLE <table_name> ... USING DELTA TBLPROPERTIES('delta.feature.catalogManaged'='supported');build/sbt generate by @yili-db in #1231Full Changelog: v0.3.0...v0.3.1
The 0.3.1 release of Unity Catalog AI includes important bug fixes, improved error handling, and enhanced reliability for function execution and client connections.
🔧 Improved Function Client Defaults: Fixed default client configuration in get_uc_function_client to automatically set up DatabricksFunctionClient when Databricks environment is available, eliminating unnecessary warnings during toolkit initialization.
🛡️ Enhanced SQL NULL Handling: Improved handling of SQL NULL default parameters in both Databricks and OSS clients, ensuring proper validation and execution when NULL values are used as function parameter defaults.
🔄 Better Connection Recovery: Enhanced retry handling logic for DatabricksFunctionClient with improved session expiration detection patterns, including support for INVALID_HANDLE errors and more robust reconnection notifications.
🐞 Smarter Spark Session Management: Optimized Spark session creation to only initialize when needed - sessions are now created on-demand for function creation rather than during client initialization in local execution mode.
📋 Improved Error Messages: Added better error handling for Pydantic validation errors with user-friendly messages, particularly for SQL NULL parameter issues and type validation failures.
🔧 Dependency Version Updates: Set maximum version constraint for databricks-connect to <16.4 to ensure compatibility with serverless compute environments.
📚 Enhanced Documentation: Added comprehensive development loop tutorial notebook and updated integration documentation with corrected examples and improved usage patterns.
Update log4j version due to vulnerability with 2.23.1 and build failure. by @creechy in https://github.com/unitycatalog/unitycatalog/pull/878
We are excited to announce Unity Catalog 0.3.0 release which includes exciting new features and many bug-fixes and improvements.
A huge thank you to the awesome community who made this release possible!
Artifacts in this release:
UnityCatalog now works seamlessly with the recently released 4.0 versions of Spark and DeltaLake.
This release adds two new API surfaces - credentials and external locations. The addition of these new securables lays the groundwork for much more flexible handling of external storage services (S3, Blob, etc.). In future releases Tables, Volumes, Models, and all storage based securables will then be able to fully leverage this new flexibility while fully integrated with Unity Catalog’s security model.
The community worked hard on providing a new deployment option using an official Helm chart making it simple to run UnityCatalog in Kubernetes based environments. As part of this effort we are now releasing a separate UI image to serve the UC UI component.
To allow users to always test the latest features added to Unity Catalog, we are now publishing development images of the server and ui container on all merges to main.
pip install daft by @ccmao1130 in https://github.com/unitycatalog/unitycatalog/pull/976Full Changelog: https://github.com/unitycatalog/unitycatalog/compare/v0.2.1...v0.3.0
We are excited to announce Unity Catalog 0.3.0 release which includes exciting new features and many bug-fixes and improvements.
A huge thank you to the awesome community who made this release possible!
Artifacts in this release:
UnityCatalog now works seamlessly with the recently released 4.0 versions of Spark and DeltaLake.
This release adds two new API surfaces - credentials and external locations. The addition of these new securables lays the groundwork for much more flexible handling of external storage services (S3, Blob, etc.). In future releases Tables, Volumes, Models, and all storage based securables will then be able to fully leverage this new flexibility while fully integrated with Unity Catalog’s security model.
The community worked hard on providing a new deployment option using an official Helm chart making it simple to run UnityCatalog in Kubernetes based environments. As part of this effort we are now releasing a separate UI image to serve the UC UI component.
To allow users to always test the latest features added to Unity Catalog, we are now publishing development images of the server and ui container on all merges to main.
pip install daft by @ccmao1130 in #976Full Changelog: v0.2.1...v0.3.0
We are excited to announce Unity Catalog 0.2 which adds a lot of exciting new features.
We are excited to announce Unity Catalog 0.2 which adds a lot of exciting new features.
Artifacts in this release:
For more information on the UC Roadmap, please refer to Proposed UC Roadmap CY2024Q4 #411
This release has been made possible by 63 contributors with 31 new contributors as highlighted below. Thank you to everyone for their support and for making this release possible.
https://github.com/unitycatalog/unitycatalog/compare/v0.1.0...v0.2.0
uc command by @creechy in https://github.com/unitycatalog/unitycatalog/pull/142quickstart.md by @thomhart31 in https://github.com/unitycatalog/unitycatalog/pull/183pre-commit and markdown formatter with mdformat by @FredrikBakken in https://github.com/unitycatalog/unitycatalog/pull/322/self endpoint. by @creechy in https://github.com/unitycatalog/unitycatalog/pull/428sbt generate by @vikrantpuppala in https://github.com/unitycatalog/unitycatalog/pull/474javafmtAll on main repo by @vikrantpuppala in https://github.com/unitycatalog/unitycatalog/pull/479Remove Deprecated Jackson Property Naming Strategy by @ledbutter in https://github.com/unitycatalog/unitycatalog/pull/94
We are excited to announce Unity Catalog 0.1, the first release of the Unity Catalog (UC) open source project! This release gives a preview of the following exciting new features.
Artifacts in this release:
This is the metastore server that supports REST APIs (Open API specification) for cataloging all different data and AI assets. In version 0.1, the server has the REST APIs to support the following:
Catalogs and Schemas
Tables for storing tabular, structured data
Volumes for storing non-tabular datasets (unstructured data)
Functions - List, get, and create functions for Python functions (AI/ML workloads) and SQL functions
See documentation more details
This Java SDK generated from the OpenAPI specification operates with any UC compliant with the Unity REST API.
Though not part of this release, this is an example UC connector that demonstrates how to use the UC SDK to operate on various data assets. Specifically, it has the following functionality
See documentation more details
This release has been made possible by 33 contributors. Thank you to everyone for their support and for making this release possible.
copybara.py file by @fpgmaas in https://github.com/unitycatalog/unitycatalog/pull/6vv typo in banner splash by @rmoff in https://github.com/unitycatalog/unitycatalog/pull/90mkdocs by @fpgmaas in https://github.com/unitycatalog/unitycatalog/pull/32Full Changelog: https://github.com/unitycatalog/unitycatalog/commits/v0.1.0
Nothing published for this version
Nothing published for this version
Your coding agent can read these notes before it upgrades. Set up the MCP server →