NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #4299 most downloaded on PyPI
Official Python SDK for Unity Catalog
Last release 1 months ago
20 Aug 2026
Release timing varies
gaps range from 5 weeks to 6 months
Nearly every release is documented
notes for 8 of 8 stable releases
Nothing withdrawn
no release was ever pulled
2 years old
10 releases · first in 2024
One column per month.
Breaking Changes and Migration Notes
We are thrilled to announce Unity Catalog 0.6.0 release, which includes exciting new features, many bug fixes, and improvements. A huge thank you to the awesome community who made this release possible!
Unity Catalog 0.6.0 adds a connector artifact for Apache Spark 4.2.x, alongside the existing Spark 4.0.x and 4.1.x artifacts. This is the artifact that unlocks the view and metric-view DDL described below.
| Spark Versions | UC Spark connector artifact |
|---|---|
| Apache Spark 4.0.x | unitycatalog-spark_4.0_2.13 |
| Apache Spark 4.1.x | unitycatalog-spark_4.1_2.13 |
| Apache Spark 4.2.x | unitycatalog-spark_4.2_2.13 |
To initialize a Spark session for Spark 4.2.x, use the following command:
export CATALOG_NAME=unity
export UC_URI=http://localhost:8080
export UC_TOKEN=<uc-token>
bin/spark-sql --name "unity-spark42" \
--master "local[*]" \
--packages "org.apache.hadoop:hadoop-aws:3.4.2,io.delta:delta-spark_4.2_2.13:4.4.0,io.unitycatalog:unitycatalog-spark_4.2_2.13:0.6.0" \
--conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \
--conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" \
--conf "spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem" \
--conf "spark.sql.catalog.$CATALOG_NAME=io.unitycatalog.spark.UCSingleCatalog" \
--conf "spark.sql.catalog.$CATALOG_NAME.uri=$UC_URI" \
--conf "spark.sql.catalog.$CATALOG_NAME.token=$UC_TOKEN" \
--conf "spark.sql.defaultCatalog=$CATALOG_NAME"For other Spark versions, use the same command with the matching --packages coordinates:
org.apache.hadoop:hadoop-aws:3.4.1,io.delta:delta-spark_4.0_2.13:4.4.0,io.unitycatalog:unitycatalog-spark_4.0_2.13:0.6.0org.apache.hadoop:hadoop-aws:3.4.2,io.delta:delta-spark_4.1_2.13:4.4.0,io.unitycatalog:unitycatalog-spark_4.1_2.13:0.6.0NOTICE: the UC Spark connector 0.6.0 works with Delta Lake 4.4.0, and uses the UC Delta API to access UC managed tables by default.
Metric views give you a governed semantic layer: define reusable named aggregations (measures) and the columns used to slice them (dimensions) once in Unity Catalog, and every consumer computes them from the same definition instead of hand-writing the aggregation in each query.
Unity Catalog 0.5.0 introduced the catalog side of this as an experimental preview — the server could store and serve metric views. 0.6.0 now fully supports metric views with Apache Spark 4.2: you define them in SQL, query their measures in SQL, and Unity Catalog governs them like any other object.
Unity Catalog 0.6.0 also adds SQL views (VIEW) as a catalog object. A view is a named, governed SQL query: the server stores the view's SQL text, its column list, and the tables it reads from, so any engine can discover and resolve it through the standard /tables API surface.
Engine support: read on Spark 4.0 / 4.1, create on Spark 4.2. Creating a view requires Spark's v2 view-catalog API, which landed in Spark 4.2. On Spark 4.0 and 4.1 the connector cannot create, replace, rename, or drop a view — but it can read one:
| Capability | Spark 4.0 / 4.1 | Spark 4.2 |
|---|---|---|
| CREATE VIEW, DROP VIEW | not supported | supported |
| SELECT, DESCRIBE | supported | supported |
| Listing | via SHOW TABLES | via SHOW VIEWS and SHOW TABLES |
Replacing a view (CREATE OR REPLACE VIEW) and renaming one (ALTER VIEW ... RENAME TO) are not supported yet on any version.
The UC Delta API introduced in 0.5.0 gains the remaining table-lifecycle operations, so a Delta engine can now operate catalog-managed Delta tables through one REST surface.
This release includes several hardening fixes in credential vending and authorization. All are server-side; no configuration change is required.
Wildcard issuer and audience allowlists. server.allowed-issuers and server.audiences now accept * as a single-DNS-label wildcard, which makes multi-tenant deployments practical without enumerating every tenant host.
Configurable access token lifetime. UC access tokens issued by the /auth/tokens token exchange now carry an expiration, configurable via server.access-token-timeout (default PT24H), with the lifetime reported back to clients in the response's expires_in field. See the breaking-change notice below.
Configurable OAuth scope. The client-credentials OAuth flow previously hardcoded scope=all-apis, which is correct for the UC/Databricks token endpoint but invalid for other identity providers. You can now set the scope explicitly.
UC access tokens now expire. Before 0.6.0, UC access tokens issued by the /auth/tokens token exchange had no expiration and were valid indefinitely. Starting with 0.6.0 they expire after server.access-token-timeout, which defaults to 24 hours.
expires_in value now returned by /auth/tokens) at least once per day under the default setting.server.access-token-timeout=PT720H
auth.accessTokenTimeout.etc/conf/token.txt is a service token and is unchanged — it still does not expire.Full Changelog: v0.5.1...v0.6.0
Thanks to everyone who contributed to this release:
Anas Khan, Bhavya Joshi, Chen Wang, Matt Van Horn, Murali Ramanujam, Scott Haines, Simon Maling, Thomas Wiest, Timothy Wang, Yi Li, Yuya Ebihara, foss-contributor, openinx, seewishnew, SreeramaYeshwanthGowd, tiennguyen-onehouse
We are pleased to announce the Unity Catalog 0.5.1 release, a patch release focused on credential-caching reliability and performance, plus authorizat
We are pleased to announce the Unity Catalog 0.5.1 release, a patch release focused on credential-caching reliability and performance, plus authorization fixes for permission APIs. A huge thank you to the awesome community who made this release possible!
Artifacts in this release:
Unity Catalog 0.5.1 improves how vended cloud credentials are cached and shared across queries in the unitycatalog-hadoop module (used by the UC Spark connector and other Hadoop-based engines).
Previously, each query could trigger a fresh credential fetch even when accessing the same table or path, because the global cache was keyed by a one-off identifier generated per request. Starting with 0.5.1, credentials are cached by their natural scope — the catalog resource they apply to — so repeated reads or writes against the same UC-managed table or path reuse a valid vended credential instead of round-tripping to the UC server on every query.
Key improvements:
These changes apply automatically to Spark users on the credential-scoped file system introduced in 0.5.0. No new configuration is needed.
Unity Catalog 0.5.1 fixes the permission GET endpoints so they work correctly when server-side authorization is enabled (#1603).
Before this fix, calling GET routes on the permissions API — such as listing effective permissions on a catalog, schema, table, or other securable — could return HTTP 500 errors when authentication and authorization were turned on. With 0.5.1, these endpoints authenticate and authorize requests as expected, making it straightforward to inspect permissions on secured deployments.
Affected routes include permission lookups across metastore, catalog, schema, table, and other securable types exposed by the permissions REST API.
Full Changelog: https://github.com/unitycatalog/unitycatalog/compare/v0.5.0...v0.5.1
Yi Li, openinx
We are thrilled to announce Unity Catalog 0.5.0 release, which includes exciting new features, many bug fixes, and improvements. A huge thank you to t
We are thrilled to announce Unity Catalog 0.5.0 release, which includes exciting new features, many bug fixes, and improvements.
A huge thank you to the awesome community who made this release possible!
Artifacts in this release:
Unity Catalog 0.5.0 introduces a dedicated UC Delta API (/api/2.1/unity-catalog/delta/v1/…) that lets any Delta client create, load, list, alter, rename, and delete catalog-managed Delta tables through a single, standardized REST surface.
The new UC Delta API enables Delta engines such as Spark, Flink, Trino, DuckDB, and others to use Unity Catalog as a versioned, Delta-native centralized catalog, while preserving the server-side validation needed to protect catalog-managed tables from unsafe writes.
Key features:
/config, giving clients a stable contract and allowing the API to evolve without breaking certified versions.catalogs/{catalog}/schemas/{schema}/tables/{table}).Here is the quick start with curl command:
# 1. start the server
bin/start-uc-server
# 2. set the base path
export UC="http://localhost:8080/api/2.1/unity-catalog/delta/v1"
# 3. check the API is up
curl -s "$UC/config?catalog=unity&protocol-versions=1.0" | jq
# 4. load a preloaded table's metadata
curl -s "$UC/catalogs/unity/schemas/default/tables/numbers" | jq
# 5. vend read credentials for it
curl -s "$UC/catalogs/unity/schemas/default/tables/numbers/credentials?operation=READ" | jq
The 0.5.0 release introduces distinct artifacts for the UC Spark connector to provide native and robust support across different Spark versions. By separating these artifacts, the connector can be better tailored to the specific APIs of each Spark version.
Artifact mapping:
| Spark Versions | UC Spark connector artifact |
|---|---|
| Apache Spark 4.0.x | unitycatalog-spark_4.0_2.13 |
| Apache Spark 4.1.x | unitycatalog-spark_4.1_2.13 |
To initialize a Spark session for Spark 4.0.x, use the following command:
export CATALOG_NAME=unity
export UC_URI=http://localhost:8080
export UC_TOKEN=<uc-token>
bin/spark-sql --name "unity-spark40" \
--master "local[*]" \
--packages "org.apache.hadoop:hadoop-aws:3.4.0,io.delta:delta-spark_4.0_2.13:4.3.0,io.unitycatalog:unitycatalog-spark_4.0_2.13:0.5.0" \
--conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \
--conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" \
--conf "spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem" \
--conf "spark.sql.catalog.$CATALOG_NAME=io.unitycatalog.spark.UCSingleCatalog" \
--conf "spark.sql.catalog.$CATALOG_NAME.uri=$UC_URI" \
--conf "spark.sql.catalog.$CATALOG_NAME.token=$UC_TOKEN" \
--conf "spark.sql.defaultCatalog=$CATALOG_NAME"
For Spark 4.1.x, utilize this command:
export CATALOG_NAME=unity
export UC_URI=http://localhost:8080
export UC_TOKEN=<uc-token>
bin/spark-sql --name "unity-spark41" \
--master "local[*]" \
--packages "org.apache.hadoop:hadoop-aws:3.4.2,io.delta:delta-spark_4.1_2.13:4.3.0,io.unitycatalog:unitycatalog-spark_4.1_2.13:0.5.0" \
--conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \
--conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog" \
--conf "spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem" \
--conf "spark.sql.catalog.$CATALOG_NAME=io.unitycatalog.spark.UCSingleCatalog" \
--conf "spark.sql.catalog.$CATALOG_NAME.uri=$UC_URI" \
--conf "spark.sql.catalog.$CATALOG_NAME.token=$UC_TOKEN" \
--conf "spark.sql.defaultCatalog=$CATALOG_NAME"
NOTICE: this uc spark connector 0.5.0 will need to work with delta 4.3.0, and it will use the brand new UC Delta API to access the UC managed table by default.
In 0.4.1, the credential-scoped file system (which prevents the long-running-session out-of-memory error described in #1378) required explicitly setting spark.sql.catalog.<catalog-name>.credScopedFs.enabled=true. Starting with 0.5.0, the credential-scoped file system is enabled by default (#1479): long-running Spark sessions that touch many UC-managed tables across different storage locations are protected out of the box.
If you need to disable the credential-scoped file system for any reason, you can explicitly opt out:
spark.sql.catalog.<catalog-name>.credScopedFs.enabled=false
Unity Catalog 0.5.0 adds experimental support for Metric Views as a first-class catalog object — a semantic layer over one or more underlying tables, where you define reusable aggregations as measures and the dimensions used to slice them at query time.
The definition is supplied as free-form text in view_definition — typically a YAML semantic spec, but a SQL statement works too; Unity Catalog stores it as-is and leaves interpretation to the query engine.
The YAML body declares the source table, the dimensions to group by, and the measures to aggregate:
version: "0.1"
source: unity.default.source_events
dimensions:
- name: event_day
expr: date_trunc('day', event_time)
measures:
- name: event_count
expr: count(*)
In this release you can create, list, load, and drop metric views via the UC REST API and the Java client. A metric view is another TableType (METRIC_VIEW) on the existing /tables surface, carrying a view_definition (the YAML or SQL body above) and a non-empty view_dependencies list, all inspectable through standard getTable / listTables calls.
Example: creating a metric view via the UC REST API:
curl -X POST 'http://localhost:8080/api/2.1/unity-catalog/tables' \
-H 'Content-Type: application/json' \
-d '{
"name": "daily_event_counts",
"catalog_name": "unity",
"schema_name": "default",
"table_type": "METRIC_VIEW",
"view_definition": "version: \"0.1\"\nsource: unity.default.source_events\ndimensions:\n - name: event_day\n expr: date_trunc('\''day'\'', event_time)\nmeasures:\n - name: event_count\n expr: count(*)",
"view_dependencies": {
"dependencies": [
{ "table": { "table_full_name": "unity.default.source_events" } }
]
}
}'
Unity Catalog 0.5.0 introduces a new Maven module: io.unitycatalog:unitycatalog-hadoop.
This module extracts UC cloud credential providers and the credential-scoped file system from the Spark connector into a reusable Hadoop-based library. As a result, engines like Flink, Trino, and others can now reuse UC-vended cloud credentials and credential renewal without depending on Spark.
It includes:
Configuration, so engine authors do not need to manually translate UC credentials into cloud-specific Hadoop settingsStarting with 0.5.0, the UC Spark connector depends on this new unitycatalog-hadoop module.
Full Changelog: https://github.com/unitycatalog/unitycatalog/compare/v0.4.1...v0.5.0
Chen Wang, ChengJi, Lukas Reining, Murali Ramanujam, Robert Pack, Scott Haines, Serena Ruan, Siddharth Murching, Timothy Wang, Vladan Vasić, Yi Li, Yuki Watanabe, Zhen Li, Zheng Hu, schen2401, Vishnu Chandrashekhar
Nothing published for this version
Breaking change: Existing deployments with authorization enabled must add both properties to server.properties before upgrading:
We are thrilled to announce Unity Catalog 0.4.1 release, which includes exciting new features and many bug fixes, and improvements.
A huge thank you to the awesome community who made this release possible!
Artifacts in this release:
Unity Catalog 0.4.1 introduces a credential-scoped file system for the UC Spark connector, fixing an out-of-memory error that could occur in long-running Spark sessions accessing many UC-managed tables across different storage locations.
This feature is disabled by default. To enable it, add the following to your Spark configuration:
spark.sql.catalog.<catalog-name>.credScopedFs.enabled=true
In Unity Catalog 0.4.1, we’ve expanded atomic write guarantees (for UC Managed Delta Tables) to include:
REPLACE TABLE / REPLACE TABLE AS SELECT (RTAS)DPO)Before UC 0.4.0, these operations could leave a table in a partially updated state if a job failed mid-execution. In UC 0.4.0, we added a fast-fail safeguard to prevent these unsafe writes.
With UC 0.4.1, these operations are now fully atomic:
This ensures stronger reliability and eliminates the risk of corrupted or partially written tables during overwrite operations.
How to use
To enable this functionality, run the Unity Catalog Spark connector with Delta Lake 4.2.0.
Support the VARIANT data type in the UC API specification. With this, you can now create and access a table of the VARIANT type via the UC client. Take the UC Apache Spark connector as an example (Apache Spark™ 4.1.0 or above is required):
CREATE TABLE data_table (
id INT,
event_data VARIANT -- Stores JSON-like structures
) USING DELTA
TBLPROPERTIES (
'delta.feature.catalogManaged' = 'supported'
);
INSERT INTO data_table (id, event_data) VALUES
(1, PARSE_JSON('{"type": "login", "timestamp": "2026-03-25T10:00:00Z"}')),
(2, PARSE_JSON('{"type": "logout", "user_id": 123, "duration_seconds": 60}'));
Prior to 0.4.1, the UC server did not constrain which identity provider a token could come from -- any token signed by its own issuer's key would pass validation, allowing complete user impersonation. This release adds mandatory issuer and audience validation when authorization is enabled. Reported and fixed by @lukas-reining.
server.allowed-issuers: Comma-separated list of trusted token issuers. Tokens from unknown issuers are rejected before JWKS fetch.server.audiences: Comma-separated list of expected audience values. Tokens not intended for this UC instance are rejected.This change also adds ECDSA key support (ES256/384/512) and proper JWT exception handling.
Breaking change: Existing deployments with authorization enabled must add both properties to server.properties before upgrading:
server.allowed-issuers=https://accounts.google.com
server.audiences=your-client-id
Cheng Ji, Lukas Reining, Robert Pack, Scott Haines, Timothy Wang, Vladan Vasić, Yi Li, Zhen Li, Zheng Hu
Full Changelog: https://github.com/unitycatalog/unitycatalog/compare/v0.4.0...v0.4.1
We are excited to announce Unity Catalog 0.4.0 release which includes exciting new features and many bug-fixes and improvements. A huge thank you to t
We are excited to announce Unity Catalog 0.4.0 release which includes exciting new features and many bug-fixes and improvements.
A huge thank you to the awesome community who made this release possible!
Artifacts in this release:
Unity Catalog server now provides full support for Storage Credentials for AWS — a first-class resource for managing cloud storage authentication. Storage credentials allow you to securely register and manage cloud provider credentials that Unity Catalog uses to govern access to the data. For example, UC will generate temporary credentials (time limited, and scoped down to a table/model/volume root directory) when an authorized user needs access to the data.
For AWS, Unity Catalog's supported credential type is the AWS IAM Role. Unity Catalog uses AWS Security Token Service ( STS) to assume this specified IAM role, which allows it to obtain temporary, scoped credentials for accessing AWS S3 cloud storage.
Managing credentials via UC CLI:
# Create a storage credential
bin/uc credential create --name my-s3-credential --aws-iam-role-arn arn:aws:iam::123456789012:role/my-data-role
# List all credentials
bin/uc credential list
# Update a credential
bin/uc credential update --name my-s3-credential --aws-iam-role-arn arn:aws:iam::123456789012:role/new-role
# Delete a credential (use --force to delete even if referenced by external locations)
bin/uc credential delete --name my-s3-credential --force
External locations are Unity Catalog objects that link a cloud storage path (for example, AWS S3) to a storage credential. They work together to enable secure, governed, and configurable access to cloud storage by vending temporary credentials instead of embedding secrets in table or volume definitions.
External locations can be used for creating any objects using cloud storage. It can be used as managed storage locations of catalogs and schemas which are used for creating MANAGED tables. It can also be used for creating EXTERNAL tables.
For each storage access, Unity Catalog automatically resolves the matching external location and vends short-lived, scoped credentials to the compute engine (for example, Spark). This centralizes credential management and ensures secure, temporary access to cloud storage.
Managing external locations via CLI:
# Create an external location
bin/uc external-location create \
--name my-s3-data \
--url s3://my-bucket/data \
--credential-name my-s3-credential
# List external locations
bin/uc external-location list
# Update an external location
bin/uc external-location update --name my-s3-data --url s3://my-bucket/new-data
# Delete an external location
bin/uc external-location delete --name my-s3-data
Unity Catalog server now supports setting a managed storage location when creating catalogs and schemas. When configured, managed Delta tables (experimental) and volumes created under that catalog or schema will automatically use storage under the parent’s managed storage location, making managed data placement consistent and easier to control.
Creating catalogs and schemas with Managed Storage Location via UC CLI:
# Create a catalog with managed storage location
bin/uc catalog create --name my_cat --storage_root s3://my-bucket/path
# Create a schema with managed storage location
bin/uc schema create --catalog some_other_cat --name my_schema --storage_root s3://my-bucket/path
In 0.3.1, automatic credential renewal for cloud storage (S3, Azure, GCS) required explicitly setting spark.sql.catalog.<catalog-name>.renewCredential.enabled=true. Starting with 0.4.0, credential renewal is enabled by default. Long-running Spark jobs will automatically renew storage credentials before they expire, preventing interruptions.
If you need to disable automatic renewal, you can explicitly set:
spark.sql.catalog.<catalog-name>.renewCredential.enabled=false
UC 0.4 now supports both Delta 4.1 with both Spark 4.0 and Spark 4.1. See Delta 4.1 release notes for more details on multiple Spark version support.
Before UC 0.4.0, CREATE TABLE AS SELECT (CTAS), REPLACE TABLE (RT), REPLACE TABLE AS SELECT (RTAS), CREATE OR REPLACE TABLE (CORT), and Dynamic Partition Overwrite (DPO) were non-atomic operations. If a failure occurred during data write or table replacement, the target table could be partially written, corrupted, or see existing data dropped.
Starting with UC 0.4.0 (used with Delta 4.1.0), Unity Catalog leverages Spark StagingTableCatalog API to enforce safer behavior:
A new unitycatalog-dspy package enables you to use Unity Catalog AI functions as tools within DSPy workflows. This allows LLM agents orchestrated by DSPy to call your registered UC functions as tools.
pip install unitycatalog-dspy
from unitycatalog.ai.dspy.toolkit import UCFunctionToolkit
import dspy
# Create a toolkit from your UC functions
toolkit = UCFunctionToolkit(
function_names=["catalog.schema.my_function"],
client=uc_client
)
# Use in a DSPy ReAct agent
agent = dspy.ReAct(
signature="question -> answer",
tools=toolkit.tools,
max_iters=5
)
Full Changelog: https://github.com/unitycatalog/unitycatalog/compare/v0.3.1...v0.4.0
The 0.4.0 release of Unity Catalog AI includes several improvements, including expanded Databricks client options and dependency/security improvements.
Warehouse ID Support in DatabricksFunctionClient: For workspaces where serverless compute is unavailable or restricted, DatabricksFunctionClient now accepts an optional warehouse_id parameter to dispatch function execution via the Statement Execution API. Serverless remains the default and recommended execution path.
Python 3.10+ Required: The minimum supported Python version has been raised to 3.10, as Python 3.9 has reached end-of-life.
unitycatalog-langchain by @bbqiu in #1120warehouse_id support to DatabricksFunctionClient by @smurching in #1428ai/core Python deps by @serena-ruan in #1457unitycatalog-ai dependency (remove redundant pydantic check, move pandas import) by @serena-ruan in #1519databricks-connect for Databricks integration by @aravind-segu in #1062unitycatalog-langchain and unitycatalog-openai package versions by @serena-ruan in #1515We are excited to announce Unity Catalog 0.4.0 release which includes exciting new features and many bug-fixes and improvements.
A huge thank you to the awesome community who made this release possible!
Artifacts in this release:
Unity Catalog server now provides full support for Storage Credentials for AWS — a first-class resource for managing cloud storage authentication. Storage credentials allow you to securely register and manage cloud provider credentials that Unity Catalog uses to govern access to the data. For example, UC will generate temporary credentials (time limited, and scoped down to a table/model/volume root directory) when an authorized user needs access to the data.
For AWS, Unity Catalog's supported credential type is the AWS IAM Role. Unity Catalog uses AWS Security Token Service ( STS) to assume this specified IAM role, which allows it to obtain temporary, scoped credentials for accessing AWS S3 cloud storage.
Managing credentials via UC CLI:
# Create a storage credential
bin/uc credential create --name my-s3-credential --aws-iam-role-arn arn:aws:iam::123456789012:role/my-data-role
# List all credentials
bin/uc credential list
# Update a credential
bin/uc credential update --name my-s3-credential --aws-iam-role-arn arn:aws:iam::123456789012:role/new-role
# Delete a credential (use --force to delete even if referenced by external locations)
bin/uc credential delete --name my-s3-credential --forceExternal locations are Unity Catalog objects that link a cloud storage path (for example, AWS S3) to a storage credential. They work together to enable secure, governed, and configurable access to cloud storage by vending temporary credentials instead of embedding secrets in table or volume definitions.
External locations can be used for creating any objects using cloud storage. It can be used as managed storage locations of catalogs and schemas which are used for creating MANAGED tables. It can also be used for creating EXTERNAL tables.
For each storage access, Unity Catalog automatically resolves the matching external location and vends short-lived, scoped credentials to the compute engine (for example, Spark). This centralizes credential management and ensures secure, temporary access to cloud storage.
Managing external locations via CLI:
# Create an external location
bin/uc external-location create \
--name my-s3-data \
--url s3://my-bucket/data \
--credential-name my-s3-credential
# List external locations
bin/uc external-location list
# Update an external location
bin/uc external-location update --name my-s3-data --url s3://my-bucket/new-data
# Delete an external location
bin/uc external-location delete --name my-s3-dataUnity Catalog server now supports setting a managed storage location when creating catalogs and schemas. When configured, managed Delta tables (experimental) and volumes created under that catalog or schema will automatically use storage under the parent’s managed storage location, making managed data placement consistent and easier to control.
Creating catalogs and schemas with Managed Storage Location via UC CLI:
# Create a catalog with managed storage location
bin/uc catalog create --name my_cat --storage_root s3://my-bucket/path
# Create a schema with managed storage location
bin/uc schema create --catalog some_other_cat --name my_schema --storage_root s3://my-bucket/pathIn 0.3.1, automatic credential renewal for cloud storage (S3, Azure, GCS) required explicitly setting spark.sql.catalog.<catalog-name>.renewCredential.enabled=true. Starting with 0.4.0, credential renewal is enabled by default. Long-running Spark jobs will automatically renew storage credentials before they expire, preventing interruptions.
If you need to disable automatic renewal, you can explicitly set:
spark.sql.catalog.<catalog-name>.renewCredential.enabled=false
UC 0.4 now supports both Delta 4.1 with both Spark 4.0 and Spark 4.1. See Delta 4.1 release notes for more details on multiple Spark version support.
Before UC 0.4.0, CREATE TABLE AS SELECT (CTAS), REPLACE TABLE (RT), REPLACE TABLE AS SELECT (RTAS), CREATE OR REPLACE TABLE (CORT), and Dynamic Partition Overwrite (DPO) were non-atomic operations. If a failure occurred during data write or table replacement, the target table could be partially written, corrupted, or see existing data dropped.
Starting with UC 0.4.0 (used with Delta 4.1.0), Unity Catalog leverages Spark StagingTableCatalog API to enforce safer behavior:
A new unitycatalog-dspy package enables you to use Unity Catalog AI functions as tools within DSPy workflows. This allows LLM agents orchestrated by DSPy to call your registered UC functions as tools.
pip install unitycatalog-dspyfrom unitycatalog.ai.dspy.toolkit import UCFunctionToolkit
import dspy
# Create a toolkit from your UC functions
toolkit = UCFunctionToolkit(
function_names=["catalog.schema.my_function"],
client=uc_client
)
# Use in a DSPy ReAct agent
agent = dspy.ReAct(
signature="question -> answer",
tools=toolkit.tools,
max_iters=5
)Note truncated.
We are excited to announce Unity Catalog 0.3.1 release which includes exciting new features and many bug-fixes and improvements.
We are excited to announce Unity Catalog 0.3.1 release which includes exciting new features and many bug-fixes and improvements.
A huge thank you to the awesome community who made this release possible!
Artifacts in this release:
This release introduces a reusable unitycatalog-client Java module that provides an abstraction layer for building Unity Catalog API clients across different integrations:
Map<String, String> configs = Map.of("type", "static", "token", "my-unitycatalog-token");
ApiClient client = ApiClientBuilder.create()
.uri("http://localhost:8080")
.tokenProvider(TokenProvider.create(configs))
.retryPolicy(JitterDelayRetryPolicy.builder().maxAttempts(5).build())
.build();
Pluggable Authentication: Introduced TokenProvider interface with built-in support for user-provided tokens and OAuth configurations, and also with support for custom dynamically-loaded providers via specifying the class name (see TokenProvider documentation for more details).
Retry and Resiliency: Standardized retry handling with RetryPolicy interface for automatic transient failure recovery and set a jittered backoff retry policy as the default to help prevent excessive request pressure on the Unity Catalog server.
The unitycatalog-spark connector now supports automatic renewal of S3/Azure/GCS storage credentials. This feature ensures that long-running Spark jobs and Unity Catalog workloads can access cloud storage without interruption:
org.apache.hadoop:hadoop-aws:3.4.0 to spark external libraries and set the extra spark configurations (example):spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
spark.sql.catalog.<catalog-name>.renewCredential.enabled=true
org.apache.hadoop:hadoop-azure:3.3.6 to spark external libraries and set the extra spark configurations (example):spark.sql.catalog.<catalog-name>.renewCredential.enabled=true
com.google.cloud.bigdataoss:gcs-connector:3.0.2 to spark external libraries and set the extra spark configurations (example):spark.sql.catalog.<catalog-name>.renewCredential.enabled=true
The Unity Catalog Spark connector now supports OAuth authentication. When OAuth is enabled, the client automatically requests short-lived tokens using the configured OAuth endpoint, client ID, and client secret. It also detects token expiration and transparently renews tokens without interrupting user workloads. To enable it, set the following to your spark configurations:
spark.sql.catalog.<catalog-name>.auth.type=oauth
spark.sql.catalog.<catalog-name>.auth.oauth.uri=<oauth-uri>
spark.sql.catalog.<catalog-name>.auth.oauth.clientId=<oauth-client-id>
spark.sql.catalog.<catalog-name>.auth.oauth.clientSecret=<oauth-client-secret>
This release introduces experimental (work‑in‑progress) support for UC‑managed Delta tables, gated by the catalogManaged table feature (see Delta RFC for more details). A UC-managed Delta table has its storage location allocated and its commits coordinated by Unity Catalog server.
Staging Table API: A new API (/staging-tables) is introduced to allow creation of a staging table and allocation of a managed location that a managed table will be built on top of.
Delta Commit API: A new Delta commit API endpoint (/delta/preview/commits, both GET and POST) is introduced to coordinate Delta commits by UC server.
UCSingleCatalog Spark integration: Now it supports creation and data read & write of a managed Delta table from Delta-Spark with storage location allocated by UC server and commits coordinated by unity catalog server.
Note: This new feature is experimental and the APIs are subject to change in future release. We are releasing this for every one to start building integrations and connectors. To test this out, you can do the following:
Start the UC server with the server property server.managed-table.enabled=true.
Start Spark 4.0 with the upcoming Delta 4.0.1 release.
An example Spark SQL to create a managed table:
CREATE TABLE <table_name> ... USING DELTA TBLPROPERTIES('delta.feature.catalogManaged'='supported');
build/sbt generate by @yili-db in https://github.com/unitycatalog/unitycatalog/pull/1231Full Changelog: https://github.com/unitycatalog/unitycatalog/compare/v0.3.0...v0.3.1
We are excited to announce Unity Catalog 0.3.1 release which includes exciting new features and many bug-fixes and improvements.
A huge thank you to the awesome community who made this release possible!
Artifacts in this release:
This release introduces a reusable unitycatalog-client Java module that provides an abstraction layer for building Unity Catalog API clients across different integrations:
Map<String, String> configs = Map.of("type", "static", "token", "my-unitycatalog-token");
ApiClient client = ApiClientBuilder.create()
.uri("http://localhost:8080")
.tokenProvider(TokenProvider.create(configs))
.retryPolicy(JitterDelayRetryPolicy.builder().maxAttempts(5).build())
.build();Pluggable Authentication: Introduced TokenProvider interface with built-in support for user-provided tokens and OAuth configurations, and also with support for custom dynamically-loaded providers via specifying the class name (see TokenProvider documentation for more details).
Retry and Resiliency: Standardized retry handling with RetryPolicy interface for automatic transient failure recovery and set a jittered backoff retry policy as the default to help prevent excessive request pressure on the Unity Catalog server.
The unitycatalog-spark connector now supports automatic renewal of S3/Azure/GCS storage credentials. This feature ensures that long-running Spark jobs and Unity Catalog workloads can access cloud storage without interruption:
org.apache.hadoop:hadoop-aws:3.4.0 to spark external libraries and set the extra spark configurations (example):spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
spark.sql.catalog.<catalog-name>.renewCredential.enabled=true
org.apache.hadoop:hadoop-azure:3.3.6 to spark external libraries and set the extra spark configurations (example):spark.sql.catalog.<catalog-name>.renewCredential.enabled=true
com.google.cloud.bigdataoss:gcs-connector:3.0.2 to spark external libraries and set the extra spark configurations (example):spark.sql.catalog.<catalog-name>.renewCredential.enabled=true
The Unity Catalog Spark connector now supports OAuth authentication. When OAuth is enabled, the client automatically requests short-lived tokens using the configured OAuth endpoint, client ID, and client secret. It also detects token expiration and transparently renews tokens without interrupting user workloads.
To enable it, set the following to your spark configurations:
spark.sql.catalog.<catalog-name>.auth.type=oauth
spark.sql.catalog.<catalog-name>.auth.oauth.uri=<oauth-uri>
spark.sql.catalog.<catalog-name>.auth.oauth.clientId=<oauth-client-id>
spark.sql.catalog.<catalog-name>.auth.oauth.clientSecret=<oauth-client-secret>
This release introduces experimental (work‑in‑progress) support for UC‑managed Delta tables, gated by the catalogManaged table feature (see Delta RFC for more details). A UC-managed Delta table has its storage location allocated and its commits coordinated by Unity Catalog server.
Staging Table API: A new API (/staging-tables) is introduced to allow creation of a staging table and allocation of a managed location that a managed table will be built on top of.
Delta Commit API: A new Delta commit API endpoint (/delta/preview/commits, both GET and POST) is introduced to coordinate Delta commits by UC server.
UCSingleCatalog Spark integration: Now it supports creation and data read & write of a managed Delta table from Delta-Spark with storage location allocated by UC server and commits coordinated by unity catalog server.
Note: This new feature is experimental and the APIs are subject to change in future release. We are releasing this for every one to start building integrations and connectors. To test this out, you can do the following:
Start the UC server with the server property server.managed-table.enabled=true.
Start Spark 4.0 with the upcoming Delta 4.0.1 release.
An example Spark SQL to create a managed table:
CREATE TABLE <table_name> ... USING DELTA TBLPROPERTIES('delta.feature.catalogManaged'='supported');build/sbt generate by @yili-db in #1231Full Changelog: v0.3.0...v0.3.1
The 0.3.1 release of Unity Catalog AI includes important bug fixes, improved error handling, and enhanced reliability for function execution and client connections.
🔧 Improved Function Client Defaults: Fixed default client configuration in get_uc_function_client to automatically set up DatabricksFunctionClient when Databricks environment is available, eliminating unnecessary warnings during toolkit initialization.
🛡️ Enhanced SQL NULL Handling: Improved handling of SQL NULL default parameters in both Databricks and OSS clients, ensuring proper validation and execution when NULL values are used as function parameter defaults.
🔄 Better Connection Recovery: Enhanced retry handling logic for DatabricksFunctionClient with improved session expiration detection patterns, including support for INVALID_HANDLE errors and more robust reconnection notifications.
🐞 Smarter Spark Session Management: Optimized Spark session creation to only initialize when needed - sessions are now created on-demand for function creation rather than during client initialization in local execution mode.
📋 Improved Error Messages: Added better error handling for Pydantic validation errors with user-friendly messages, particularly for SQL NULL parameter issues and type validation failures.
🔧 Dependency Version Updates: Set maximum version constraint for databricks-connect to <16.4 to ensure compatibility with serverless compute environments.
📚 Enhanced Documentation: Added comprehensive development loop tutorial notebook and updated integration documentation with corrected examples and improved usage patterns.
Update log4j version due to vulnerability with 2.23.1 and build failure. by @creechy in https://github.com/unitycatalog/unitycatalog/pull/878
We are excited to announce Unity Catalog 0.3.0 release which includes exciting new features and many bug-fixes and improvements.
A huge thank you to the awesome community who made this release possible!
Artifacts in this release:
UnityCatalog now works seamlessly with the recently released 4.0 versions of Spark and DeltaLake.
This release adds two new API surfaces - credentials and external locations. The addition of these new securables lays the groundwork for much more flexible handling of external storage services (S3, Blob, etc.). In future releases Tables, Volumes, Models, and all storage based securables will then be able to fully leverage this new flexibility while fully integrated with Unity Catalog’s security model.
The community worked hard on providing a new deployment option using an official Helm chart making it simple to run UnityCatalog in Kubernetes based environments. As part of this effort we are now releasing a separate UI image to serve the UC UI component.
To allow users to always test the latest features added to Unity Catalog, we are now publishing development images of the server and ui container on all merges to main.
pip install daft by @ccmao1130 in https://github.com/unitycatalog/unitycatalog/pull/976Full Changelog: https://github.com/unitycatalog/unitycatalog/compare/v0.2.1...v0.3.0
We are excited to announce Unity Catalog 0.3.0 release which includes exciting new features and many bug-fixes and improvements.
A huge thank you to the awesome community who made this release possible!
Artifacts in this release:
UnityCatalog now works seamlessly with the recently released 4.0 versions of Spark and DeltaLake.
This release adds two new API surfaces - credentials and external locations. The addition of these new securables lays the groundwork for much more flexible handling of external storage services (S3, Blob, etc.). In future releases Tables, Volumes, Models, and all storage based securables will then be able to fully leverage this new flexibility while fully integrated with Unity Catalog’s security model.
The community worked hard on providing a new deployment option using an official Helm chart making it simple to run UnityCatalog in Kubernetes based environments. As part of this effort we are now releasing a separate UI image to serve the UC UI component.
To allow users to always test the latest features added to Unity Catalog, we are now publishing development images of the server and ui container on all merges to main.
pip install daft by @ccmao1130 in #976Full Changelog: v0.2.1...v0.3.0
We are excited to announce Unity Catalog 0.2.1 release which has a few bug fixes and a new Python client - https://pypi.org/project/unitycatalog-clien
We are excited to announce Unity Catalog 0.2.1 release which has a few bug fixes and a new Python client - https://pypi.org/project/unitycatalog-client/
Artifacts in this release:
Key improvements are as follows:
warehouse configuration property for better compatibility with Iceberg connectorsFull Changelog: https://github.com/unitycatalog/unitycatalog/compare/v0.2.0...v0.2.1
Nothing published for this version
Your coding agent can read these notes before it upgrades. Set up the MCP server →