Added

• Add generic OpenLineage context configuration for propagating parent-run info #4682 @kacpermuda
  Adds one JSON context payload for parent-run info, replacing 6 separate config keys in dbt and Spark.
• Dbt: Capture incremental-strategy config and run-wide full_refresh #4716 @himakolavennu
  Captures incremental model settings, such as strategy and partitioning, plus a run-wide full_refresh flag.
• dbt: Emit exposures as a dataset facet on dbt model datasets #4705 @himakolavennu
  Adds exposures, such as dashboards and notebooks, as a dataset facet, so lineage now includes table-to-exposure links.
• Flink: Allow disabling checkpoint tracking to reduce excessive RUNNING events #4744 @wangxiaojing
  Adds a setting to turn off checkpoint tracking, cutting unnecessary RUNNING events and REST calls.
• GreatExpectations: Implement FileSizeExpectationsParser for file asset validation #4698 @maitraymukeshkumarmodi-aiml
  Implements file size checks, so file size expectation results now produce a fileSize facet instead of being ignored.
• Spec: Extend ParentRunFacet with parent and root facets #4680 @kacpermuda
  Adds an optional facets field to ParentRunFacet, so producers can forward parent and root facets to child events.
• Client/Python: Add new Datadog DCs #4711 @HeroCC
  Adds the new UK1 and US2 FedRAMP Datadog sites to the transport's accepted site list.

Fixed

• Client/Java: Fix deserialization crash for facet fields literally named additionalProperties #4743 @zerafachris
  Fixes a crash when a facet field is named additionalProperties, which confused the JSON deserialiser.
• Client/Python: Don't recurse deep_merge_dicts into a non-dict value #4731 @chuenchen309
  Fixes a crash when merging a scalar config value with a dict value from another source.
• Client/Python: Fix async HTTP transport shutdown to wait for events sharing an ID #4728 @mattfaltyn
  Fixes shutdown so it waits for every event, even when several events share the same ID.
• Client/Python: Fix several async transport issues in the Datadog transport #4694 @mobuchowski
  Fixes high CPU use and a memory leak in the Datadog transport's async worker.
• Client/Python: Make with_additional_properties() facets picklable #4729 @zerafachris
  Fixes a pickling error for facets built with with_additional_properties().
• Client/Python: Match tag keys case-insensitively in both directions #4722 @chuenchen309
  Fixes tag matching so a user tag overrides an integration tag, regardless of letter case.
• Client/Python: Restore HTTP debuglevel after emit failure #4696 @hcthakur2004
  Restores the HTTP debug level after a failed emit, so it does not stay switched on for later requests.
• Common/dbt: Consume --openlineage-dbt-job-name in its --opt=value form #4730 @chuenchen309
  Fixes job name handling so --openlineage-dbt-job-name=value no longer breaks the dbt run.
• Common/dbt: Let the ~/.dbt profiles directory fallback run #4725 @chuenchen309
  Fixes the profiles directory lookup so it now falls back to ~/.dbt/ as intended.
• Common/dbt: Resolve dbt test seed inputs by alias, not logical name #4732 @chuenchen309
  Fixes seed input resolution so tests use the seed's alias instead of its logical name.
• Common/dbt: Separate the default spark port from the host #4724 @chuenchen309
  Fixes a missing colon in the default Spark port, which broke the dataset namespace.
• Common/dbt: Use the dataset namespace for the SQL query externalQuery facet #4723 @chuenchen309
  Fixes the SQL query externalQuery facet to use the dataset namespace, not a placeholder value.
• Sql: Include WHERE subquery inputs in table lineage #4752 @mattfaltyn
  Includes tables referenced only in a WHERE subquery, such as WHERE EXISTS, in table lineage.

Added

• Dbt: Capture dbt model meta/config values #4653 @mobuchowski
  Capture per-model config values (e.g. materialized, access, owner, group) and the user-defined meta map from the dbt manifest into a new dbt_model dataset facet, attached in both the legacy/local and structured-logs dbt processors.
• Spark: Add support for open source Unity Catalog (Spark 4.0) #3674 @arturowczarek
  Add a catalog handler for the open source Unity Catalog's UCSingleCatalog Spark catalog so output dataset facets are produced when using OSS Unity Catalog, which cannot reuse the existing DeltaHandler implementation.
• Spec: Extend jobTypeJobFacet to capture event emission lifecycle #4592 @jakub-moravec
  Extend the Job Type facet so producers, particularly streaming producers that emit a single event covering a time period rather than START/RUNNING/COMPLETE events, can document the lifecycle of the OpenLineage events consumers should expect.

Fixed

• Spark: Fix missing output lineage for insertInto with append mode #4676 @mishrasangeeta87
  Fix a bug where df.write.mode("append").insertInto(table) did not emit an output dataset, by extracting the target table directly from the AppendData logical plan, aligning append-mode handling with overwrite-mode.
• Spark: Fix Unity Catalog symlink metadata on Databricks #4660 @mishrasangeeta87
  Build Databricks Unity Catalog table symlinks using the fully qualified table name and a unity-catalog namespace instead of one derived from the underlying storage path, giving lineage consumers stable, catalog-qualified identifiers.
• Client/Java: Change shaded paths to avoid conflicts when using composite transport #4652 @tnazarew
  Relocate shaded classes per-transport (e.g. io.openlineage.client.transports.gcs.shaded.x) instead of to a single shared shaded package, fixing Google library version conflicts when combining gcs and gcplineage transports in a composite transport.
• Client/Java: Normalize abfs/wasb to abfss/wasbs in ObjectStorageDatasetExtractor #4661 @arturowczarek
  Treat abfs:// and wasb:// URIs the same as their TLS variants (abfss/wasbs) when deriving the object-storage namespace, since they point at the same underlying files and only differ in transport.
• Client/Java: Stop swallowing actual cause when using CompositeTransport #4666 @mobuchowski
  Preserve the underlying exception instead of only its message when a CompositeTransport emit fails, so the real cause is visible instead of being lost.
• Common: Handle trailing command options #4624 @hcthakur2004
  Avoid indexing past the end of command-line arguments when a trailing option has no value.
• Common: Preserve nullable chain input #4627 @hcthakur2004
  Avoid mutating the caller-provided chain in get_from_nullable_chain, keeping it reusable after lookup.
• GreatExpectations: Preserve datasource URL ports #4679 @hcthakur2004
  Preserve the port on a datasource URL when Great Expectations strips credentials from it.
• Python: Isolate Datadog config defaults #4586 @hcthakur2004
  Replace shared, mutable Datadog config dictionary defaults with per-instance factories so retry and async transport rule defaults aren't accidentally shared across instances.
• Python: Support file URI transport paths #4656 @hcthakur2004
  Normalize local file:// URIs before opening file transport paths so events can be appended through a file:// log path.
• Spark: Fix Iceberg handler detection to probe SparkCatalog instead of Catalog #4665 @arturowczarek
  Detect Iceberg by the presence of the iceberg-spark SparkCatalog class rather than iceberg-core's Catalog, so the Iceberg handler no longer activates in environments where iceberg-core is on the classpath without iceberg-spark.

Added

• Client/Java: Add timeout-aware run event emission support #4613 @jsingh-yelp
  Add a timeout-aware emit overload to the Java client and Transport API, allowing callers to provide a bounded wait when emitting RunEvents; Kafka transport uses the timeout to wait for producer acknowledgement before returning.
• Flink: Add support for OpenLineage events in detached Flink Session Mode #4596 @jsingh-yelp
  Enable full OpenLineage lifecycle events (RUNNING, COMPLETE, FAIL) for Flink jobs submitted in detached Session Mode by introducing OpenLineageDetachedJobStatusChangedListener that initialises the Flink job ID from the REST API, bypassing the missing JobCreatedEvent on the JobManager side.

Fixed

• Flink: Emit ABORT event for canceled jobs instead of FAIL #4615 @wangxiaojing
  Map JobStatus.CANCELED to EventType.ABORT in Flink 2 job status handling; previously all non-FINISHED terminal statuses were mapped to FAIL, causing user-canceled jobs to appear as failures in downstream OpenLineage consumers.
• Python: Fix env var facet overwriting event-supplied values #4635 @kacpermuda
  Merge client-configured environment variables into any environmentVariables facet already set by the producer instead of replacing it; event-supplied values take precedence and a warning is logged on conflict.
• Spark: Fix missing Iceberg scan report facet using scanReportSupplier #4633 @fm100
  Fix missing IcebergScanReport input dataset facet in CTAS/RTAS queries by reading the report directly from scanReportSupplier when available, rather than relying on OpenLineageMetricsReporter which is registered after the scan has already occurred.

Added

• Java: Add CassandraJdbcExtractor #4610 @matveeysv
  Add CassandraJdbcExtractor to parse Cassandra JDBC URLs according to the driver specification, enabling lineage tracking for Cassandra databases via JDBC.

Fixed

• dbt: Add missing column lineage for structured log processor #4599 @fm100
  Fix missing column lineage facet in DbtStructuredLogsProcessor when using --consume-structured-logs option by attaching the column lineage facet to the output dataset on node finished events.
• dbt: Use fully qualified job ID for externalQuery run facet for BigQuery #4591 @fm100
  Use a fully qualified job ID (with project ID and location) for the externalQueryId in the externalQuery run facet when using the dbt-bigquery adapter; also adds externalQuery run facet support when using --consume-structured-logs.
• Spark: Fix GCP Dataproc jobId when job has multiple attempts #4598 @codelixir
  Exclude the dataproc_job_attempt_timestamp tag prefix when reading the job ID from Yarn tags in the GCP Dataproc facet, preventing the attempt timestamp from being reported as the job ID on retried jobs.
• Spark: Fix Lakehouse detection mechanism #4611 @tnazarew
  Fix incorrect Spark configuration properties used in the Lakehouse Hive Catalog detection logic and extend catalog detection to V2SessionCatalogHandler used by DatasetBuilders.
• Spark: Fix Snowflake column lineage for quoted identifiers #4602 @mrpalash-amz
  Fix missing column lineage when the Snowflake Spark connector uses quoted identifiers (e.g. in AWS Glue ETL with dbtable path) by applying stripQuotes() normalization on both sides of identifier comparisons in ColumnLevelLineageBuilder and SqlCollector.

Added

• Spark: Add iceberg s3 tables catalog support #4558 @mobuchowski
  Fix incorrect Glue table symlink attachment when using S3 Tables as the Iceberg catalog by adding dedicated S3 Tables catalog support and preventing Glue fallback when the S3 Tables catalog is active.
• Spark: Add Lakehouse Hive Catalog handling #4574 @tnazarew
  Add support for the Lakehouse (formerly BigLake) Hive Catalog used by Spark jobs running on Managed Spark (formerly Dataproc).
• Spark: Add Snowflake Iceberg REST catalog (Horizon) support #4546 @adnanhemani
  Add support for the Snowflake Horizon Iceberg REST catalog, enabling the Spark integration to emit correct dataset identifiers with Snowflake namespaces for Iceberg tables managed through Snowflake's REST catalog.

Changed

• Client/Java: Upgrade Kinesis Producer Library to 1.0.7 and AWS SDK to 2.44.3 #4536 @W-Ely
  Upgrade the Amazon Kinesis Producer Library (KPL) from 0.15.12 to 1.0.7 and update AWS SDK dependencies to 2.44.3 to maintain compatibility and pick up upstream fixes.
• Python: Log if JSON parsing fails for env var #4552 @kacpermuda
  Add a warning log message when JSON parsing of OpenLineage configuration from environment variables fails, making configuration errors easier to diagnose.
• Spark: Make iceberg catalog identifiers optional #4557 @mobuchowski
  Change BaseCatalogTypeHandler.getIdentifier() to return Optional<DatasetIdentifier> instead of a nullable value, making the contract for Iceberg catalog type handlers more explicit and null-safe.
• Spark: Refactor of Dataset builder #4587 @tnazarew
  Refactor Spark integration's dataset-building logic from overloaded methods to a more flexible builder pattern, improving extensibility and maintainability.

Fixed

• Python: Detect GitHub pull request head refs #4561 @hcthakur2004
  Fix pull request number detection in the Python client to recognize refs/pull/<number>/head format (used in GitHub Actions pull_request events) in addition to the existing refs/pull/<number>/merge form.
• Spark: Fix SnowflakeCatalogTypeHandler.getIdentifier() return type #4571 @mobuchowski
  Fix SnowflakeCatalogTypeHandler.getIdentifier() to return Optional<DatasetIdentifier> as required by BaseCatalogTypeHandler, correcting a type mismatch introduced when the Snowflake Iceberg REST catalog handler was first added.
• Spark: Map sqlserver to MsSqlDialect #4556 @dbathie-wtg
  Map Spark JDBC sqlserver URLs to MsSqlDialect so SQL Server queries using bracketed identifiers ([schema].[table]) are parsed correctly during lineage extraction.

Changed

• Client/Go: Rewrite code generator #4501 @tnazarew Replace the quicktype-based text-manipulation generator with a structured pipeline that parses spec files via go-jsonschema, resolves references, and renders facet classes — enabling extension of generated classes (e.g. byool resources). Also renames Run → RunWrapper and RunInfo → Run for consistency with Job and Dataset.

Fixed

• dbt: Use correct name for fabric adapter #4542 @mobuchowski Fix extract_adapter_type lookup for the Microsoft Fabric adapter by aligning the Adapter enum name with the dbt adapter type (fabric), while keeping the fabric-warehouse OpenLineage namespace unchanged.

Added

• dbt: Add Microsoft Fabric adapter support #4485 @harels Register dbt-fabric in the supported Adapter enum and add fabric:// namespace extraction, preventing NotImplementedError on Fabric profiles and enabling lineage tracking for Microsoft Fabric datasets.
• Flink: Add ClickHouse JDBC support #4495 @och5351 Add ClickHouseJdbcExtractor to enable lineage tracking for ClickHouse JDBC connections, supporting both jdbc:clickhouse:// and jdbc:ch:// URL schemes with optional protocol prefix removal.
• Flink: Extract generic schema and config facet visitors #4506 @jsingh-yelp Add ConfigFacetVisitor and SchemaFacetVisitor for non-Table API Flink 2 datasets, enabling schema and configuration metadata extraction for DataStream API connectors beyond the Table API.
• Spec/Python: Extend DataQualityAssertionsDatasetFacet with new information #4457 @kacpermuda Extend DataQualityAssertionsDatasetFacet with additional fields (actual value, expected value, severity) aligned with TestRunFacet, bumping the facet spec to 1-1-0.

Changed

• dbt: Expose failure counts and thresholds on dbt test facets #4500 @mobuchowski Populate actual and expected fields on DataQualityAssertionsDatasetFacet and TestRunFacet with the dbt test failure count and error threshold, enabling consumers to distinguish passing tests from failures and understand configured tolerances.

Fixed

• Client/Java: Fix MySqlJdbcExtractor incorrectly includes URL database in qualified table names #4497 @och5351 Fix MySQL JDBC extractor to not prepend the URL database to already-qualified table names, preventing invalid 3-level identifiers like mydb.schema.table1 (MySQL treats DATABASE and SCHEMA as synonyms).
• dbt: Allow retry command #4520 @mobuchowski Fix dbt-ol to recognize retry as a valid dbt command so lineage is captured when re-running failed nodes.
• dbt: Emit FAIL on aggregate test event when assertions failed #4523 @mobuchowski Fix aggregate test events in the legacy consume_local_artifacts path that emitted COMPLETE (success) even when the embedded DataQualityAssertions facet showed error-severity assertion failures.
• dbt: Fix race condition in dbt parsing #4472 @mobuchowski Fix stale manifest.json from a prior invocation being loaded before dbt finishes its parse phase by subscribing to the ArtifactWritten structured log event (dbt ≥ 1.9) and lazy-loading for older versions.
• dbt: Guard against None node_unique_id in get_job_type() #4489 @hcthakur2004 Fix AttributeError: 'NoneType' object has no attribute 'startswith' crash when node_info.unique_id is absent from the event by adding an early None guard.
• dbt: Handle dbt owner list #4515 @mobuchowski Fix crash when the dbt owner metadata field is a list rather than a string, handling both single-owner and multi-owner configurations gracefully.
• dbt: Handle missing "description" key in DbtArtifactProcessor for dbt-fusion support #4499 @ichirotakami Fix KeyError in extract_dataset_data() for dbt-fusion manifests where source nodes omit the description key by using .get() with an empty string default.
• dbt: Propagate root parent through dbt-run wrapper to child events #4522 @mobuchowski Fix root=None in the parent facet for per-node events emitted via the legacy consume_local_artifacts path by propagating root_parent_* fields to dbt_run_metadata.
• dbt: Specify utf-8 encoding when opening files #4503 @hcthakur2004 Fix file reading in the dbt provider to explicitly specify UTF-8 encoding, preventing UnicodeDecodeError on systems where the default locale encoding is not UTF-8.
• Hive: Fix NPE in Parsing.buildQueryTree for 3+ way set operations #4498 @1fanwang Fix NullPointerException when processing Hive queries with 3+ way UNION ALL, INTERSECT, or EXCEPT set operations by recursively descending through intermediate QBExpr nodes instead of assuming leaf-only structure.
• Spark: Add support for SparkChangelogTable in IcebergHandler #4439 @NETIZEN-11 Fix ClassCastException when Iceberg returns SparkChangelogTable instead of SparkTable by replacing the unsafe direct cast with instanceof checks and graceful fallback handling.
• Spark: Bump httpclient5 to 5.6.1 for CVE-2026-40542 #4521 @Poojitha-R-Rao Upgrade httpclient5 dependency to 5.6.1 to address security vulnerability CVE-2026-40542.
• Spark: Fix PathUtils.reconstructDefaultLocation malformed URI for S3 paths #4505 @creazyfrog Fix incorrect use of Hadoop's 3-argument Path constructor that produced malformed S3 URIs like s3://bucket/prefix://database.db./table_name, which caused IllegalArgumentException and silently dropped all lineage events.
• Spark: Update AWS SDK to resolve CVE-2026-33871 #4427 @Poojitha-R-Rao Upgrade AWS SDK version to address security vulnerability CVE-2026-33871 (netty-codec).

Added

• Client/Go: Add Go client #4358 @tnazarew Add a new OpenLineage Go client with code generation from the spec, HTTP and GCP Lineage transport implementations, and CI integration.
• dbt: Add sourceCodeLocation facet with repo URL override and git autodetect #4398 @himakolavennu Add sourceCodeLocation job facet emission for dbt events, with repo URL configurable via --openlineage-repo-url or OPENLINEAGE_REPO_URL, falling back to git remote get-url origin autodetection. URLs are normalized to host/org/repo format.
• dbt: Emit TestRunFacet in dbt integration #4449 @mobuchowski Emit TestRunFacet from both the regular dbt processor path and the structured logs path for test result nodes, including outcome (pass/fail/warn/error), severity, and expected/actual values.
• Spec: Add pullRequestNumber field to SourceCodeLocationJobFacet #4436 @himakolavennu Add optional pullRequestNumber field to SourceCodeLocationJobFacet (spec bumped to 1-1-0) with auto-detection from CI environment variables (GITHUB_REF for GitHub Actions, CI_MERGE_REQUEST_IID for GitLab CI).
• Spec: Add TestRunFacet spec and Python client support #4448 @mobuchowski Add TestRunFacet JSON Schema for recording test outcomes (pass/fail/warn/error), severity, expected vs actual values, and test message, along with generated Python client code and facet registration.

Changed

• Client/Python: Make sourceCodeLocation opt-in and reduce subprocess overhead #4403 @himakolavennu Make git autodetect for sourceCodeLocation opt-in by default (disabled=True) to avoid surprising subprocess calls during event emission, and reduce subprocess count from 4 to 2 by consolidating git calls.
• Python: Drop Python 3.9 support, minimum version is now 3.10 #4391 @mobuchowski Remove support for Python 3.9, which reached end-of-life in November 2024. The minimum supported Python version is now 3.10.

Fixed

• Client/Java: Fix three bugs in transports-kinesis SPI registration and NPE #4451 @bahram-cdt Fix broken SPI directory path (META-INF.services/ → META-INF/services/), wrong class name in SPI file (missing .kinesis sub-package), and NullPointerException when properties config is omitted in the Kinesis transport.
• dbt: Provide data quality assertions for singular tests #4400 @mobuchowski Fix the structured logs processor to correctly resolve parent datasets for dbt singular tests using parent_map, so test result assertions are now attached to the correct datasets as DataQualityAssertionsDatasetFacet.
• Spark: Fix handling of InMemoryRelation with repeating unqualified column names #4390 @orthoxerox Fix column-level lineage for InMemoryRelation when it contains non-unique unqualified column names by using qualifiedName instead of name for correct column mapping.
• Spark: Fix Iceberg Glue catalog symlink #4384 @usamakunwar Fix dataset symlinks when using Glue catalog for Iceberg by detecting Glue-based catalogs via Spark configuration and generating correct ARNs even when not using the Hive metastore.
• Spark: Prevent alias names from appearing as phantom input fields in JDBC column lineage #4366 @mobuchowski Fix JDBC column lineage to skip extractInternalInputs when SQL-based column lineage is already available, preventing alias names from being incorrectly included as input fields alongside the original column names.

Added

• Flink: Add DatasetConfigFacet support for Flink native listener #4368 @jsingh-yelp Add support for emitting DatasetConfigFacet in the Flink native listener, enabling configuration tracking for datasets processed by Flink jobs.
• Spec: Add HierarchyDatasetFacet #3747 @dolfinus Introduce the HierarchyDatasetFacet to provide structured representation of dataset hierarchy levels (database, schema, table, etc.) without relying on dataset name parsing, enabling consistent handling across different database systems with varying hierarchy depths.

Changed

• Spark: Replace semanticHash with identity-based VisitedNodes tracking #4383 @mobuchowski Improve performance by switching from expensive semanticHash() calls to identity-based tracking using IdentityHashMap, eliminating hot path in large jobs during plan traversal.
• Spark: Use HashSet instead of LinkedList for BFS visited set #4376 @mobuchowski Optimize findDependentInputs by replacing LinkedList with HashSet for visited node tracking, improving lookup performance from O(n) to O(1).

Fixed

• dbt: Fix singular tests handling in structured logs #4372 @mobuchowski Fix handling of dbt singular tests when processing structured logs to ensure proper test result tracking and lineage extraction.
• Spark: Fix BigLake catalog detection condition #4357 @tnazarew Fix the condition for adding project_id to catalog properties, ensuring proper BigLake catalog detection and handling.

Fixed

• dbt: Attach ExtractionErrorRunFacet on metadata extraction failures #4349 @harels Attach ExtractionErrorRunFacet to run events when @handle_keyerror-decorated extraction methods fail, making previously invisible extraction errors visible to downstream consumers instead of silently emitting incomplete events.
• dbt: Fix KeyError handling in _get_model_node #4348 @harels Fix exception type mismatch in _get_model_node() by raising KeyError instead of RuntimeError, allowing the @handle_keyerror decorator to catch it and return None gracefully when a node_id is not found in the manifest.
• dbt: Use .get() for optional project version retrieval #4345 @zagoodman Fix crash when version key is absent from dbt_project.yml, which became optional in dbt 1.5, by using .get() instead of direct key access.

Added

• Client: Add JWT authentication support #4313 @jakub-moravec Add JWT authenticator for Java and Python clients, enabling token-based authentication without requiring a custom authenticator implementation.
• Spark: Extract input symlinks from DataSourceRDD #4283 @kchledowski Enable extraction of input dataset symlinks from DataSourceRDD, providing richer lineage information for RDD-based Iceberg operations.

Changed

• Spark: Disable column-level lineage for LogicalRDD plans #4329 @kchledowski Disable column-level lineage extraction for LogicalRDD plans to prevent incorrect lineage caused by lost schema and transformation context.
• Spark: Disable input schema extraction for LogicalRDD and add schema extraction for Iceberg DataSourceRDD #4331 @kchledowski Disable unreliable input schema extraction from LogicalRDD and instead extract schemas from Iceberg table metadata when reading via DataSourceRDD.

Fixed

• dbt: Align schema definition for dbt facets #4285 @LegendPawel-Marut Align schema definitions for dbt-run-run-facet and dbt-version-run-facet to fix validation inconsistencies.
• dbt: Better error handling for finding the directory of profiles.yml #4320 @ah12068 Handle missing profiles_dir key in run_results.json gracefully, falling back to default profile directory resolution.
• dbt: Fix --target-path CLI flag being ignored #4298 @gaurav-atlan Fix the --target-path CLI argument not being parsed and passed to artifact processors, causing the default target path to always be used.
• Flink: Use Flink 2.x-only class for version detection #4312 @mobuchowski Fix false Flink 2.x detection when modern V2-based connectors are used with Flink 1.x by using JobStatusChangedListenerFactory for version detection.
• Java: Send Content-Encoding header for compressed requests #4282 @Lukas-Riedel Send the Content-Encoding header when request body compression is enabled in the Java client, consistent with the Python client behavior.
• Spark: Early return in DatabricksEventFilter for non-Databricks platforms #4315 @mobuchowski Add fast environment detection check to skip Databricks-specific event filtering on non-Databricks platforms, reducing overhead.
• Spark: Handle null Glue ARN in IcebergHandler #4311 @mobuchowski Fix NullPointerException when processing Iceberg datasets with AWS Glue catalog by safely handling null Glue ARN values.
• Spark: Handle ResolvedIdentifier in DropTableVisitor #4316 @mobuchowski Fix ClassCastException on DROP TABLE commands in Databricks Runtime 14.2+ by handling ResolvedIdentifier alongside ResolvedTable.
• Python: Remove unneeded build runtime dependency #4344 @mobuchowski Remove the build package from runtime dependencies as it is only needed at build time and is already handled by the build system configuration.
• Spark: Resolve parent job name in ParentRunFacet for AWS Glue #4340 @tstrilka Fix incorrect parent job name in ParentRunFacet for child events (SQL_JOB, RDD_JOB) on AWS Glue, where the raw spark.app.name was used instead of the resolved application name from platform-specific name resolvers.
• Spark: Skip Iceberg RDD integration tests on Java 8 with Spark 3.5 @mobuchowski Skip testAppendWithRDDTransformations and testAppendWithRDDProcessing on Java 8 + Spark 3.5 where the Iceberg vendor module is not compiled due to Iceberg 1.7 requiring Java 11+.

Added

• dbt: Extract test severity from dbt tests #4258 @mobuchowski Add support for extracting and reporting severity information from dbt tests in OpenLineage events.
• Spec: Add severity field to DataQualityAssertionsDatasetFacet #4257 @mobuchowski Extend the DataQualityAssertionsDatasetFacet with a severity field to indicate the importance level of data quality assertions.
• Spark: Add support for DataSourceRDD with Iceberg #4263 @kchledowski Enable lineage extraction from DataSourceRDD when reading Iceberg data sources, supporting mixed RDD and DataFrame operations common in AWS Glue environments.

Fixed

• dbt: Fix send-events command argument parsing #4262 @mobuchowski Fix IndexError when running 'dbt-ol send-events' command without additional arguments by properly checking args length.
• Spark: Fix BigQuery column-level lineage namespace #4264 @mobuchowski Fix incorrect "namespace" value in BigQuery column-level lineage to match the format used by regular BigQuery dataset collection.
• Spark: Fix isDeltaPlan() to handle multiple Spark extensions #4268 @mobuchowski Fix Delta detection failing when multiple Spark extensions are configured (comma-separated), enabling proper event filtering in environments like Azure Fabric or when using Gluten.
• Spark: Fix META-INF/services files of relocated packages #4243 @kchledowski Fix service provider interface conflicts caused by package relocation not updating META-INF/services configuration files.

Added

• DataZone transport: Add cross-region support #4218 @RohithKayathi Enable posting lineage events to DataZone domains in different regions from where data transformation jobs run.
• Spark: Add config for disabling RDD event emitting #4118 @kchledowski Add new configuration option spark.openlineage.filter.rddEventsDisabled to selectively disable OpenLineage event emission for RDD operations while keeping SQL-based operations enabled.
• Spark: Add schema and CLL facets for Snowflake writes #4124 @kchledowski Add schema and column-level lineage support for Snowflake datasets when using the Spark-Snowflake connector.
• Spark: Add spark.openlineage.applicationRunId override #4215 @wslulciuc Add support to override the application runID via the property spark.openlineage.applicationRunId.
• Spec: Add ExecutionParametersRunFacet #4182 @jakub-moravec Add a new facet to capture input parameters supplied to a job at the time of execution, enabling reproducibility, debugging, and richer lineage context.

Changed

• Java: Update GCP Lineage transport version and fix dependency shading #3768 @tnazarew Update GCP Lineage transport to use new version of the producer library with fixed dependency shading.
• Python: Show what transport failed to create #4220 @mobuchowski Improve error messages to indicate which transport failed to create.
• Spark: Prevent classloader issue by gating log behind additional flag #4207 @mobuchowski Fix classloader conflicts with BigQuery connector by gating DEBUG toJSON() logging behind an additional flag and logging exceptions.

Fixed

• Python: Fix .with_additional_properties() annotation #4197 @dolfinus Fix type annotation for .with_additional_properties() method to correctly accept keyword arguments.
• Spark: Fix BigQuery symlinks with ".db" suffix #4192 @kchledowski Fix BigQuery symlink namespace incorrectly having ".db" suffix in RUNNING and COMPLETE events by avoiding mutation of the Identifier object.
• Spark: Fix Glue Data Catalog detection in YARN cluster mode #4229 @lawofcycles Add fallback mechanism to retrieve AWS region from EC2 Instance Metadata Service when environment variables are unavailable in YARN cluster mode.
• Spark: Fix missing inputs and CLL for AWS DynamicFrame #4222 @kchledowski Fix missing inputs and column-level lineage when writing from AWS DynamicFrame by treating NewHadoopRDD as file-like.
• Spark: Remove path pattern in ColumnLineageFacet as well #4228 @RohithKayathi Apply spark.openlineage.dataset.removePath.pattern to input field names in ColumnLineageFacet, and fix hashCode/equals methods to include additionalProperties.

Removed

• Airflow: Remove Airflow integration from OpenLineage repository #4212 @kacpermuda The deprecated Airflow integration has been removed from the OpenLineage repository.

Added

• Spec: Add arbitrary extra info to JobDependency in JobDependenciesRunFacet #4189 @kacpermuda Add support for arbitrary extra information in JobDependency within JobDependenciesRunFacet.
• Python: Add debug mode to file transport #4185 @kacpermuda Add debug mode support to file transport for better troubleshooting.
• dbt: Add dbt model meta.owner to OpenLineage events #4160 @harels Add support for capturing dbt model owner information from meta.owner in OpenLineage events.
• dbt: Add DbtNodeJobFacet #4151 @mobuchowski Add DbtNodeJobFacet to provide additional dbt node information in job facets.
• Spark: Add default name to hive catalog facet #4161 @tnazarew Add default name support to Hive catalog facet in Spark integration.
• Spark: Fetch input statistics for single input RDD #4134 @pawel-big-lebowski Add support for fetching input statistics for single input RDD jobs.

Changed

• SQL: Migrate from sqlparser fork to upstream 0.59 #4153 @kchledowski Migrate SQL parser from fork to upstream version 0.59 for better maintenance and compatibility.
• Spark: Less aggressive normalization in UUID case #4178 @mobuchowski Reduce aggressiveness of UUID normalization in Spark integration.

Fixed

• Python: Small log change #4186 @kacpermuda Improve logging output in Python client.
• Spark: Ensure relation size in bytes is sane #4165 @dolfinus Fix relation size calculation to ensure values are within reasonable bounds.
• Spec: Add missing job facet schema #4154 @mobuchowski Add missing job facet schema to specification.

Fixed

• Python: re-add missing version variables in top of releaseable modules #4135 @mobuchowski Fixes breaking change in version 1.40.0.

Added

• Spec: standardize batch API endpoint #4109 @jakub-moravec Add a standardized batch API endpoint to OpenLineage specification for handling multiple events in a single request.
• Spec: Add ordinal position to SchemaDatasetFacet #4116 @mobuchowski Add ordinal_position field to track the position of fields in schema (1-indexed).
• Spec: Add JobDependenciesRunFacet #4112 @kacpermuda Introduce JobDependenciesRunFacet to track dependencies between jobs.
• Spec: Add support for temporary datasets #4103 @jakub-moravec Add support for temporary datasets to enable job-to-job lineage tracking.
• Spark: Add fallback for BigQuery project ID configuration #4075 @luke-hoffman1 Add fallback configuration for BigQuery project ID in Metastore integration.
• Spark: Add COALESCE transformation support #4123 @kacpermuda Include examples in Python generated classes for better documentation.
• Java: Add support for jTDS JDBC URL format #4077 @dolfinus Add support for parsing jTDS JDBC URL format in Java client.
• Hive: Add ParentRunFacet #4066 @tnazarew Add ParentRunFacet to Hive integration for tracking parent-child run relationships.
• Hive: Add LOAD and IMPORT handling #4097 @tnazarew Add support for tracking LOAD and IMPORT operations in Hive.
• Hive: Add EXPORT handling #4085 @tnazarew Add support for tracking EXPORT operations in Hive.
• Hive: Add START event emission #4079 @tnazarew Add START event emission support to Hive integration.

Fixed

• Spark: Fix dataset facet builders for inputs #4121 @usamakunwar Fix Spark dataset facet builders for input datasets.
• Spark: Fix job name trimming #4114 @kchledowski Fix job name trimming logic in Spark integration.
• Spark: Fix putAll on immutable maps #4113 @pawel-big-lebowski Fix putAll operation failing on immutable maps.
• Spark: Fix RDD job handling #4108 @pawel-big-lebowski Fix multiple issues with RDD job handling in Spark.
• Spark: Fix JDBC dbtable parsing #4102 @kchledowski Fix JDBC dbtable parsing to support any FROM clauses.
• Spark: Fix Databricks setup #4083 @pawel-big-lebowski Fix Spark connector configuration for Databricks environments.
• Spark: Catch NoClassDefFoundError #4099 @mobuchowski Catch NoClassDefFoundError when buggy implementations exist on classpath.
• Spark: Fix Snowflake identifier parsing #4104 @mobuchowski Fix Snowflake identifier parsing to handle quoted identifiers correctly.
• Spark: Fix Snowflake account name handling #4105 @mobuchowski Strip quotes from Snowflake account names for proper handling.
• Spec: Fix facet property names #4092 @fm100 Fix facet property names from snake_case to camelCase for consistency.
• Python: Fix facet generator after UV migration #4111 @kacpermuda Fix Python client facet generator after moving to UV build system.
• Python: Fix retry config merge #4093 @antonlin1 Fix retry configuration default merge with user-defined config in HTTP transports.
• Java: Fix CVE in commons-lang3 #4084 @mandalbalmukund Upgrade commons-lang3 version to fix CVE security vulnerability.
• Hive: Generate same runId for START and STOP events #4126 @dolfinus Ensure START and STOP events share the same runId in Hive integration.

Added

• Spark: Normalize dataset names with configurable trimmers #3996 @pawel-big-lebowski Add configurable dataset name normalization with support for date patterns, key-value pairs, and S3 location detection to enable proper dataset subsetting.
• Spark: Add missing facets in inputs for Databricks Unity Catalog #4057 @kchledowski Add missing input symlink facets for Databricks Unity Catalog tables.

Changed

• Spark: Refactor tests for dependency collector #4058 @kchledowski Refactor column-level lineage dependency collector tests for better organization and maintainability.

Fixed

• Spec: Fix typo in iceberg commit report facet spec file #4069 @fm100 Fix typo in IcebergCommitReportOutputDatasetFacet property name.
• Spark: Fix dataset trimming for CLL inputs #4061 @pawel-big-lebowski Fix dataset name trimming for column-level lineage inputs.
• Python: Remove numpy import #4062 @kacpermuda Remove unnecessary numpy import from Python client.

Removed

• Dagster: Remove Dagster integration #3844 @kacpermuda Remove Dagster integration from the repository.

Added

• Spec: Add subset dataset facets to spec #4008 @pawel-big-lebowski Add subset dataset facets to OpenLineage specification for representing dataset relationships.
• Spec: Add DatasetQualityMetricsDatasetFacet #3978 @heron-- Allow attaching dataset quality information outside of InputDatasetFacet.
• Spark: Add support for microbatch source write #4018 @tnazarew Add support for Spark structured streaming microbatch source write operations.
• Spark: Add catalog properties to catalog facet #4016 @ddebowczyk92 Add catalog properties support to Spark integration for better catalog metadata tracking.
• Spark: Add GCP project ID and location to BigQuery Metastore catalog properties #4039 @ddebowczyk92 Enhance BigQuery integration with GCP project ID and location in catalog properties.
• Spark: Add support for COALESCE transformation #3972 @kchledowski Add support for tracking COALESCE transformations in Spark jobs.
• Spark: Add catalog facet when using vanilla Hive tables #3982 @ddebowczyk92 Add catalog facet support for vanilla Hive table operations.
• Spark: Make output statistics available within complete event #4013 @pawel-big-lebowski Output statistics now available in complete events for better observability.
• Spark: Add output stats for RDD jobs #3977 @pawel-big-lebowski Add output statistics tracking for Spark RDD-based jobs.
• Java: Add equals and hashcode methods into generated classes #4050 @pawel-big-lebowski Improve generated model classes with proper equals and hashcode implementations.
• dbt: Capture dbt tags #4022 @mobuchowski Add support for capturing dbt tags in OpenLineage events.
• dbt: Add dbt Cloud account ID to DbtRunRunFacet #4017 @mobuchowski Add dbt Cloud account ID tracking to dbt run facets.
• dbt: Update DbtRunRunFacet to add more useful information #3987 @mobuchowski Enhance DbtRunRunFacet with additional metadata for better observability.
• Python: Add GCP Lineage transport #4006 @ddebowczyk92 Add native Google Cloud Platform Lineage transport for Python client.
• Python: Add fsspec support for FileTransport #3983 @JDarDagran Add fsspec filesystem support to FileTransport for broader filesystem compatibility.
• Python: Add default tags with OL client version #3980 @kacpermuda Automatically add OpenLineage client version as default tag in events.
• Airflow: Add GCP Composer facets #3986 @gabrysiaolsz Add GCP Cloud Composer environment metadata facets to Airflow integration.

Changed

• dbt: Use alias when naming datasets #4055 @mobuchowski Use dbt model aliases when generating dataset names for more accurate lineage.
• Spark: Serialize event to JSON for logging #4029 @EugeneYushin Serialize OpenLineage events to JSON format for improved debug logging.
• Spark: Respect overridden appName in EventEmitter #4030 @EugeneYushin Properly respect user-overridden application names in event emission.
• Spark: Refactor CLL ExpressionDependencyCollector #4003 @kchledowski Refactor column-level lineage expression dependency collector for better maintainability.
• Spark: Improve logging in IcebergInputStatisticsInputDatasetFacetBuilder #3994 @JDarDagran Enhance logging for Iceberg input statistics collection.
• Spark: Limit external getFileStatus calls when dealing with lots of S3 objects #3985 @pawel-big-lebowski Optimize S3 operations by limiting external getFileStatus calls for large object sets.
• Java/Spark/Hive: Move TransformationInfo to Java client to reuse across integrations #3964 @kchledowski Refactor TransformationInfo into shared Java client for cross-integration reuse.
• Python: Improve logging in AsyncHttpTransport #4026 @dolfinus Enhance logging capabilities in asynchronous HTTP transport.
• Python: Allow type aliases #4000 @JDarDagran Support Python type aliases in client code generation.
• Python: Fix classes generation for almost identical classes #3997 @JDarDagran Improve code generation to properly handle nearly identical class definitions.
• Python: Raise errors if custom token provider cannot be loaded #4014 @dolfinus Fail fast with clear errors when custom token providers fail to load.
• Python: Don't silence import errors in DefaultTransportFactory #4015 @dolfinus Improve error visibility by not silencing import errors in transport factory.
• Python: Import from facet_v2 and event_v2 instead of generated modules #3968 @kacpermuda Update import paths to use versioned facet and event modules.
• Java: Refactor ExecutorService management in OpenLineageClientUtils #4012 @JDarDagran Improve thread pool management in Java client utilities.
• CI: Replace pre-commit with prek across CI and documentation #3965 @JDarDagran Migrate from pre-commit to prek for pre-commit hook management.

Fixed

• Spark: Fix false Hive Glue detection #4053 @jsjasonseba Fix incorrect Glue catalog detection due to always attempting ARN resolution.
• Spark: Fix CLL on hiveless runtimes #4052 @kchledowski Fix column-level lineage failures on Spark runtimes without spark-hive package.
• Spark: Fix missing inputs and CLL on some table creation commands #4031 @kchledowski Fix missing input datasets and column-level lineage for CreateDataSourceTableAsSelect and CreateHiveTableAsSelect commands.
• Spark: Rely on BQ bucket info inside BigQueryIntermediateJobFilter #4044 @EugeneYushin Fix BigQuery intermediate job filtering by using bucket configuration.
• Spark: Fix for TypeNotPresentException/RefreshTableCommand errors in Spark 3.0.2 #4002 @MaciejGajewski Add additional exception handling for TypeNotPresentException in Spark 3.0.2.
• Python: Fix license field in pyproject.toml when using build module #4034 @JDarDagran Correct license field specification in Python package metadata.
• Python: Accept both apikey and api_key in token provider #4045 @kacpermuda Support both naming conventions for API key configuration parameter.
• Java: Fix empty sources jar generation #4037 @EugeneYushin Fix build issue causing empty sources JAR files to be generated.

Added

• Python: Add Datadog transport with configurable async routing #3950 @mobuchowski
  Add Datadog transport with intelligent routing between sync/async transports based on configurable rules. Supports wildcard matching and provides seamless integration with Datadog's observability platform.
• Spark: Implement support for WriteDelta, WriteIcebergDelta logical plan nodes #3860 @orthoxerox
  Add support for WriteDelta and WriteIcebergDelta logical plan nodes in Spark integration.
• dbt: Add option to override dbt job name #3933 @mobuchowski
  Add configuration option to override dbt job names in OpenLineage events.
• Java: Add Jackson Blackbird module for JSON performance optimization #3923 @kyungryun
  Improve JSON serialization performance with Jackson Blackbird module.

Changed

• Spark: Remove Spark 2 support #3904 @pawel-big-lebowski
  Drop support for Spark 2.x versions. Minimum supported version is now Spark 3.x.
• Python: Change gzip compression level in HTTP transport #3956 @dolfinus
  Optimize HTTP transport performance by adjusting gzip compression level.
• Spark: Add support for Spark 4 in streaming tests #3925 @SalvadorRomo
  Extend streaming integration tests to support Spark 4.0.

Fixed

• Spark: Improve performance of column level lineage #3946 @pawel-big-lebowski
  Limit memory consumption, provide limits for the amount of dependencies processed (1M) and input fields returned in the facet (100K). Turns on dataset lineage by default.
• Spark: Add schema size limit for column level lineage processing #3949 @ddebowczyk92
  Add limits to prevent performance issues with large schemas in column-level lineage processing.
• Spark: Fix context factory for Spark 4 #3934 @pawel-big-lebowski
  Fix context factory implementation for Spark 4.0 compatibility.
• Spark: Fix LogicalRelation constructor compatibility for Spark 4 #3930 @yunchipang
  Fix LogicalRelation constructor to maintain compatibility with Spark 4.0.
• Spark: Fix vendors parsing in SparkOpenLineageConfig #3947 @ddebowczyk92
  Fix parsing of vendor configurations in Spark OpenLineage configuration.
• dbt: Use correct namespace for dbt externalQuery facet #3953 @jroachgolf84
  Fix namespace handling in dbt external query facets.
• Python: Fix tags configuration #3943 @JDarDagran
  Fix configuration handling for user-supplied tags in Python client.

Added

• Spark: support Delta 4.0 and cover it with tests on Spark 4.0. #3877 @pawel-big-lebowski
  Fix failing tests for Spark 4.0. Make delta integration tests pass with Delta 4.0 on Spark 4.
• Spark: Add memory info to debug facet. #3914 @pawel-big-lebowski
  Extend DebugFacet with additional information on Spark's driver memory configuration and current memory usage.
• Spark: Add new AlterTableCommandDatasetBuilder for Spark 4.0. #3921 @pawel-big-lebowski
  Add support for AlterTableCommand dataset building in Spark 4.0.
• dbt: Add query IDs for dbt. #3890 @jroachgolf84
  Add query ID tracking to dbt integration.
• dbt: Add query ID capture in structured logs. #3918 @mobuchowski
  Capture query IDs from dbt structured logs for better traceability.
• Python: Formalize dataset naming for Python client. #3816 @ddebowczyk92
  Formalize dataset naming conventions in Python client implementation.

Changed

• Spark: bump minor versions 3.4.3 -> 3.4.4, 3.5.4 -> 3.5.6. #3907 @pawel-big-lebowski
  Bump tested Spark versions.
• Spark: Close OpenLineageClient in onApplicationEnd. #3851 @dolfinus
  Ensure proper cleanup of OpenLineageClient when Spark application ends.
• Python: Do not use f-strings with logging module. #3895 @dolfinus
  Replace f-string usage in logging calls with proper logging formatting.
• Python: Update protobuf version to be compatible with newer libraries. #3899 @Shadi
  Update protobuf dependency to maintain compatibility with newer library versions.
• Website: Documentation for compatibility tests. #3869 @mobuchowski
  Add documentation explaining compatibility testing processes.

Fixed

• Spark: make visitors stateless - avoid memory leak. #3902 @pawel-big-lebowski
  Merge SqlExecutionRDDVisitor and LogicalRDDVisitor classes to avoid memory leak.
• Spark: refactor iceberg handler. #3909 @pawel-big-lebowski
  Refactor Iceberg handler implementation for better maintainability.
• Spark: retry exception on empty row. #3908 @pawel-big-lebowski
  Add retry logic for handling empty row exceptions.
• Spark: fix Spark version for databricks test. #3911 @pawel-big-lebowski
  Fix Spark version configuration in Databricks test environment.
• Flink: Fix connector of type kafka-upsert not identifying kafka topics correctly. #3915 @fetta
  Fix kafka-upsert connector to properly identify kafka topics.
• Airflow: Fail fast and reduce timeout for airflow tests. #3905 @kacpermuda
  Improve test performance by implementing fail-fast behavior and reduced timeouts.
• dbt: more telemetry, fix quadratic file reading. #3916 @mobuchowski
  Improve telemetry collection and fix performance issues with file reading.
• dbt: Fix dbt version. #3894 @mobuchowski
  Fix dbt version compatibility issues.
• Python: Fix filenames for windows users. #3889 @pawel-big-lebowski
  Fix filename handling to work correctly on Windows systems.
• Transport: Adjust log level when aliasing default_http transport. #3897 @dolfinus
  Adjust logging level for transport aliasing messages.
• Build: Improve comments and add some tests. #3901 @kacpermuda
  Improve code documentation and add additional test coverage.

Added

• Spark: Include spark_applicationDetails facet to all events #3848 @dolfinus
  Add spark_applicationDetails facet to all OpenLineage events emitted by the Spark integration
• Spark: Support additional facets #3850 @ddebowczyk92
  Adds support for additional facets in Spark integration
• Spark: disable connector by Spark config parameter #3880 @pawel-big-lebowski
  Add spark.openlineage.disabled entry to disable OpenLineage integration through Spark config parameters
• Spark: Fine-grained timeout config #3779 @pawel-big-lebowski
  Add extra timeout options to emit incomplete OpenLineage events in case of timeout when building facets. See buildDatasetsTimePercentage and facetsBuildingTimePercentage in docs for more details
• Python: Asynchronous HTTP transport implementation #3812 @mobuchowski
  Adds high-performance asynchronous HTTP transport with event ordering guarantees, configurable concurrency, and comprehensive error handling. Features START-before-completion event ordering, bounded queues, and real-time statistics
• dbt: Add DbtRun facet to dbt run events #3764 @dolfinus
  Adds DbtRun facet for tracking dbt run information
• Python: Add continue_on_success and sorting transport in CompositeTransport #3829 @kacpermuda
  Adds configuration options for CompositeTransport to control behavior and ordering
• Hive: Add jobType facet #3789 @dolfinus
  Adds jobType facet to Hive integration
• Hive: Add dialect=hive to SqlJobFacet #3863 @dolfinus
  Adds dialect field to SqlJobFacet for Hive integration
• Spec: SqlJobFacet now contains dialect #3819 @mobuchowski
  Adds dialect field to SqlJobFacet specification
• Spec: Formalize job naming #3826 @ddebowczyk92
  Formalizes job naming conventions in the specification
• Spec: Formalize dataset naming #3775 @ddebowczyk92
  Formalizes dataset naming conventions in the specification

Changed

• Spark: Use Hive as default Iceberg catalog implementation #3858 @dolfinus
  Updates Spark integration to use Hive as the default catalog implementation for Iceberg tables
• Spark: Replace weak hash-map with a map with weak keys and entries #3856 @ddebowczyk92
  Improves memory management in Spark integration by replacing weak hash-map implementation
• Spark: Support latest databricks runtime #3811 @pawel-big-lebowski
  Updates Spark integration to support the latest Databricks runtime
• Python: Remove transport.wait_for_completion() #3881 @dolfinus
  Removes wait_for_completion() method from Python transport interface
• Python: Reuse session in sync HttpTransport #3843 @dolfinus
  Improves performance by reusing HTTP sessions in synchronous transport
• Python: Implement Transport.close() for Datazone and Kinesis #3857 @dolfinus
  Adds proper cleanup methods for Datazone and Kinesis transports
• Python: Implement TransformTransport.close #3855 @dolfinus
  Adds cleanup method for TransformTransport
• Python: Implement KafkaTransport.wait_for_completion() and .close() #3838 @dolfinus
  Adds proper cleanup and completion methods for Kafka transport
• Java: Make CompositeTransport.close() more reliable #3841 @dolfinus
  Improves reliability of CompositeTransport cleanup process
• Java: Cover OpenLineageClient.close() with tests #3839 @dolfinus
  Adds test coverage for OpenLineageClient cleanup methods
• Java: Name threads used in Java client #3817 @mobuchowski
  Adds meaningful names to threads used in Java client for better debugging
• Flink: Close OpenLineageClient in onJobExecuted #3854 @dolfinus
  Ensures proper cleanup of OpenLineageClient in Flink 1.x integration
• Flink: Fixed a bug incorrectly loading configuration in Event Emitter #3799 @pan-siekierski
  Fixes configuration loading issue in Flink Event Emitter
• dbt: Make invocation_id field optional #3796 @dolfinus
  Makes invocation_id field optional in dbt integration
• dbt: More resiliency for missing dbt nodes #3836 @mobuchowski
  Improves error handling for missing dbt nodes
• Hive: Add docker-compose example for local testing #3800 @dolfinus
  Adds docker-compose setup for local Hive integration testing
• Airflow: Send pending events after Airflow DAG is finished #3849 @dolfinus
  Ensures all pending events are sent after DAG completion in Airflow integration

Fixed

• dbt: Fix log path, more precise file reading #3793 @mobuchowski
  Improves log file handling in dbt integration
• dbt: Fix deprecated configs #3859 @kacpermuda
  Replaces deprecated dbt configurations with current alternatives
• Spark: Fix missing .db suffix in database/namespace location name for BigQueryMetastoreCatalog #3874 @ddebowczyk92
  Fixes database naming issue in BigQuery Metastore catalog implementation
• Spark: Fix missing output's dataset catalog facet when running CTAS queries on Iceberg tables #3835 @ddebowczyk92
  Fixes missing catalog facet in output datasets for CTAS queries on Iceberg tables
• Spark: Delta merge with column #3871 @pawel-big-lebowski
  Fixes Delta merge operation handling with column-level lineage
• Spark: Support delta 3.3.2 #3861 @pawel-big-lebowski
  Adds support for Delta Lake version 3.3.2
• Spark: Call version utils method when dataset can be identified #3832 @pawel-big-lebowski
  Fixes version handling when dataset can be properly identified
• Java: Raise error if no events were emitted by composite transport #3853 @kacpermuda
  Adds error handling when CompositeTransport fails to emit any events
• Java: Fix IntelliJ reload #3825 @mobuchowski
  Fixes IntelliJ project reload issues
• Java: Fix spotless in hive integration #3806 @mobuchowski
  Fixes code formatting issues in Hive integration
• Java: Fix missing field in SQL facet test #3830 @mobuchowski
  Fixes missing field in SQL facet test case
• Java: Fix composite transport logging of multiple transports #3887 @kacpermuda
  Fixes logging issue when using CompositeTransport with multiple transports
• Rust: Fix new rust formatting warnings #3814 @mobuchowski
  Fixes formatting warnings in Rust code

Added

• Hive: Integration added. #3555 @tnazarew with @ddebowczyk92, @jphalip
  Added OpenLineage Hive integration
• Spark: Support dynamic frames. #3691 @pawel-big-lebowski
  Support lineage extraction from UnionRdd and NewHadoopRDD, which makes dynamic frames docker based test passing.
• Hive: Add hive_query facet #3781 @dolfinus
  Adds hive_query facet for Hive integration
• Hive: Add job sql facet #3777 @dolfinus
  Adds job sql facet for Hive integration
• Hive: Add hive session facet #3786 @dolfinus
  Adds hive session facet for Hive integration
• Java: Add Location Symlink type #3717 @tnazarew
  Add new symlink type representing physical location of dataset
• Spark: Smart debug facet. #3715 @pawel-big-lebowski
  Automatically turn on debug facet in case of spark connector anomalies detected
• Spark: Add support for Big Query Metastore catalog type #3760 @ddebowczyk92
  Adds support for BigQuery Metastore catalog in Spark integration
• dbt: Add DbtRun facet #3738 @dolfinus
  Adds DbtRun facet for tracking dbt run information
• dbt: Initial support for Clickhouse #3739 @dolfinus
  Adds initial support for ClickHouse in dbt integration
• dbt: Add processing_engine facet #3725 @dolfinus
  Adds processing_engine facet for dbt integration
• Flink: Add facet with Flink jobId #3744 @dolfinus
  Adds facet containing Flink job ID information
• Flink: Add processing_engine facet #3726 @dolfinus
  Adds processing_engine facet for Flink integration
• JDBC: Column level lineage for jdbc queries load #3763 @pawel-big-lebowski
  Adds column-level lineage support for JDBC queries for Spark with single input table
• Spec: Add contentType to documentation facet #3748 @dolfinus
  Adds contentType field to documentation facet specification

Changed

• Airflow: Remove Airflow < 2.5.0 support #3669 @kacpermuda
  Drops support for Airflow versions below 2.5.0
• dbt: Use adapter rows_affected as outputStatistics #3731 @dolfinus
  Uses adapter's rows_affected for output statistics instead of custom calculation
• dbt: Move facets from processor module #3713 @dolfinus
  Refactors dbt facets organization by moving them from processor module
• Java: Speedup generateNewUUID #3754 @dolfinus
  Improves performance of UUID generation in Java client
• Java: Make UUIDUtils.generateStaticUUID random part more variative #3709 @dolfinus
  Increases randomness in static UUID generation
• Java: Add log if load from yaml fails #3766 @mvitale
  Adds logging when YAML configuration loading fails
• Spark: Update Spark 4 dependency to 4.0.0 (remove -preview1 suffix) #3751 @ddebowczyk92
  Updates Spark 4 dependency to final 4.0.0 release
• Spark: Disable module metadata file generation #3785 @ddebowczyk92
  Disables generation of module metadata files in Spark integration
• Python: Use attr.define instead of attr.s #3776 @kacpermuda
  Modernizes Python code to use newer attrs API
• Proxy: Remove native proxy #3680 @mobuchowski
  Removes the native proxy implementation

Fixed

• Spark: Fix missing table path in InsertIntoHadoopFsRelationCommand #3773 @dolfinus
  Fixes issue where table path was missing in InsertIntoHadoopFsRelationCommand
• BigQuery: Filter temp inner jobs for bigquery indirect mode #3722 @pawel-big-lebowski
  Filters out temporary inner jobs in BigQuery indirect mode
• dbt: dbt-ol should not error on job complete if there is no start event #3749 @mobuchowski
  Prevents errors when job completion occurs without corresponding start event
• Flink: Do not hide OpenLineage config parsing errors #3724 @dolfinus
  Improves error visibility for OpenLineage configuration parsing in Flink
• Java: Prevent original events from being mutated in TransformTransport #3728 @JDarDagran
  Ensures original events remain immutable during transport transformations
• Java: Fix visibility of GcpLineageTransportConfig.Mode #3762 @ngorchakova
  Corrects visibility modifier for GCP transport configuration mode

Added

• Python: add TransformTransport for Python client #3697 @kacpermuda
  Introduces the TransformTransport class for event transformations.
• Spec: add CatalogDatasetFacet #3659 @mobuchowski
  Adds specification of the CatalogDatasetFacet.
• Spark: implement CatalogDatasetFacet #3695 @mobuchowski
  Implements CatalogDatasetFacet for Spark integration.
• Transport: add Amazon DataZone transport #3685 @shinabel
  Adds support for publishing events to Amazon DataZone via dedicated transport.
• JobNamespaceReplaceTransformer: modify parent run facet #3706 @kacpermuda
  Allows modifications of the parent run facet through JobNamespaceReplaceTransformer.

Changed

• dbt: remove empty facets #3700 @dolfinus
  Prevents empty facets from being emitted by dbt integration.
• Spark: broaden exception handling for Iceberg table writes #3686 @luke-hoffman1
  Broadens exception handling to address missing column-level lineage issues.

Fixed

• dbt: generate new runId for each start event #3696 @dolfinus
  Ensures dbt start events have unique run identifiers when calling id generation methods multiple times.
• dbt: skip database if not set #3707 @dolfinus
  Ensures database field is optional and handled gracefully.
• Spark: don't swallow exceptions in getIcebergTable #3688 @dolfinus
  Improves exception visibility during Iceberg table retrieval.
• Flink 2: support JDBC naming #3681 @pawel-big-lebowski
  Correctly handles JDBC dataset naming conventions in Flink 2 integration.

Added

• Avro: support schema facet for Avro datasets #3650 @pawel-big-lebowski
  This PR adds support for schema facets in Avro datasets.
• Java: add UUIDUtils.generateStaticUUID utility #3672 @martinovm
  Provides utility method for generating static UUIDs.

Fixed

• dbt: forward dbt's return code #3682 @MassyB
  Forwards dbt's process return code correctly.
• dbt: handle dbt log file rotation #3683 @MassyB
  Implements log rotation handling to avoid log file overflow.
• Flink2: fix dataset namespace resolvers #3676 @pawel-big-lebowski
  Correctly resolves dataset namespaces in Flink2.
• Spark: ensure Spark Job Name suffix extraction #3667 @luke-hoffman1
  Adds condition and unit test to correctly extract Spark Job Name suffix.
• Spark: fix NoSuchElementException when calling get() on Option.None #3673 @ddebowczyk92
  Prevents exceptions when Option.None is encountered.
• Spark: avoid InputDataset duplicates by skipping visited nodes #3663 @ddebowczyk92
  Ensures nodes aren't revisited, avoiding duplicate InputDataset creation.

Added

• Flink: enhance JDBC extractors with additional types #3652 @HuangZhenQiu

Changed

• Python: bump supported Python version to 3.9 #3648 @mobuchowski
• Java: generate Kinesis partitionKey using Run parent facet #3637 @dolfinus

Fixed

• Core: check for RunEvent v2 in filters and env_var_facet #3664 @mudakacper
• dbt: more robust handling of dbt CommandCompleted event #3661 @mobuchowski
• Spark: define equals for TransformedInput to avoid duplicates #3644 @tnazarew
• Flink: prevent initializing Flink2 listener for Flink1 #3645 @pawel-big-lebowski
• Spark: remove RepairTableCommandVisitor from Spark3VisitorFactoryImpl #3641 @ddebowczyk92

Added

• Java: added support for LDAP connection strings #3612 @luke-hoffman1
• dbt: added support for dbt-glue adapter #3625 @mehdimld
• Python: introduced CLI for facet generator #3605 @kuba0221
• spec: optional root parent run ID added to ParentRunFacet #3572 @mobuchowski

Changed

• Java: use parent.root facet as primary source for Kafka messageKey #3622 @dolfinus
• Python: use parent.root facet as primary source for Kafka messageKey #3624 @dolfinus

Fixed

• Flink2: fix job namespace retrieval from config #3621 @dolfinus
• Spark: extract column lineage from query string #3582 @ddebowczyk92
• Spark: fix duplicate input dataset creation when RelationV2 is a child of SubqueryAlias #3607 @ddebowczyk92
• Java: fix Oracle thin JDBC URL normalization #3586 @dolfinus
• Spark: fix internal metrics sent through DebugFacet. #3594 @pawel-big-lebowski Register simple micrometer registry when no other registry configured.

Fixed

• dbt: read structured log file incrementally #3580 @MassyB Buffered writing could cause dbt to write broken log lines. This PR handles that case.
• Python: fix user tags in event v2 events #3576 @pawel-big-lebowski This PR fixes case where event v2 job and run events did not get user tags.
• Spark: fix NullPointerException #3583 @ddebowczyk92 This PR fixes potential NullPointerException in SaveIntoDataSourceCommandVisitor.

Added

• Flink 2: added support for CheckpointFacet #3531 @pawel-big-lebowski Similar to Flink 1 integration, Flink 2 integration will emit CheckpointFacet.
• Flink 2: include SQL comments in OL events #3528 @pawel-big-lebowski Table and field comments are available within generated OL events.

Changed

• Java: remove deprecated configs: 'disabledFacets' and 'timeout'. #3522 @pawel-big-lebowski Configs have been replaced with: 'facets.facet-name.disabled=true' and 'timeoutInMillis'.

Fixed

• Spark: enable Iceberg metrics reporting for SparkSessionCatalog. #3538 @sakjung Fixes support for metrics in Iceberg SparkSessionCatalog.
• Spark: inject metrics reporter to Iceberg's RESTCatalog. #3515 @pawel-big-lebowski Fixes support for metrics in Iceberg RESTCatalog.
• dbt: fix race condition on dbt log file #3535 @MassyB Fixes race condition that was happening when using structured logs output.
• dbt: fixes handling of skipped dbt nodes #3545 @MassyB Skipped nodes no longed cause exceptions.
• Spark: fix record count stats mixed with bytes stats #3550 @pawel-big-lebowski InputStatistics facet for Iceberg datasets no longer produces incorrect stats.
• Spark: catch input duplicates for SubqueryAlias. #3548 @pawel-big-lebowski Subquery alias no longer is duplicating the inputs.
• Spark: catch Java 17 add opens exception. #3552 @mobuchowski Fixes catching InaccessibleMethodException in Java 17 within SparkExtensionVisitor.

New Contributors

• @sakjung made their first contribution in https://github.com/OpenLineage/OpenLineage/pull/3538
• @noklam made their first contribution in https://github.com/OpenLineage/OpenLineage/pull/3540

Added

• Python: allow adding user-supplied tags facets from config #3471 @leogodin217 User-supplied tags will allow the client to inject new tags or override tags provided by the integrations for jobs and runs.
• Java: allow adding user-supplied tags facets from config #3493 @mobuchowski Enabled parsing tags from config in Java client and Spark conf.

Changed

• Java: change async breaker timeout setting is not a real timeout. #3487 @pawel-big-lebowski Properly name case where TaskQueueCircuitBreaker allows a configurable blocking time after submitting a callable.
• Flink: enabled circuit breaker for Flink 2 integration. #3503 @pawel-big-lebowski Native Flink integration is now isolated within circuit breaker call.

Fixed

• Spark: use all of the underlying classloaders to find META-INF/services resources. #3483 @ddebowczyk92 ServiceLoader should not fail to load OpenLineageExtensionProvider implementations in certain configurations.
• Flink: handle default null job manager address. #3486 @MarquisC Null Flink Job Manager address will default to localhost
• dbt: handle tests on sources for structured logs option. #3488 @MassyB Handle case for tests on sources which don't have the attached_node defined in the manifest.

Added

• Java: enable specifying custom SSL context #3444 @pawel-big-lebowski Enable providing configuration for SSL context within HTTP transport.
• Spark: make Spark nodes filtering configurable. #3442 @pawel-big-lebowski Spark integration filters OpenLineage events for specific plan node classes. This can be now extended with extra config entries: allowedSparkNodes and deniedSparkNodes. See Spark Configuration documentation for more details.
• Java: add task queue based async circuit breaker. #3437 @aritrabandyo This circuit breaker that executes task on a queue backed threadpool, gives up tasks if the queue is full, and keeps track of rejected tasks.
• dbt: added initial support for Trino adapter #3429 @whitleykeith This allows Trino integration to emit proper events containing Trino datasets.
• Spark: increased coverage for Spark DML commands #3430 @ssanthanam185 Adds coverage for AlterTableRecoverPartitionsCommandVisitor, RefreshTableCommandVisitor, RepairTableCommandVisitor.

Changed

• Spark: the OpenLineageSparkListener was refactored to have a public, single-argument constructor taking an instance of SparkConf. #3425 @d-m-h This presents no functional change to the listener, however it will allow for improved initialisation of the listener in the future.
• Spark: Unsupported catalog exception should be less verbose. #3435 @pawel-big-lebowski In case of unsupported classes, warn logs without a stacktrace should be produced.
• Spark: Directly expose the LogicalPlan and SparkPlan objects inside OpenLineageContext. #3443 @d-m-h This is an initial refactor to a larger code base change that will see the removal of direct access of the QueryExecution object. It has no functional change on the way the integration behaves.

Fixed

• Spark: improve column lineage by including inputs within COMPLETE events. #3434 @pawel-big-lebowski *Send input datasets in COMPLETE events while making sure version facet is attached on START only.
• dbt: ParentRunFacet is now correctly attached when using structured logs option. #3432 @MassyB Fixes incorrect structure of ParentRunFacet.

Added

• Flink: Experimental version for flink native lineage listener. #3099 @pawel-big-lebowski New flink listener to extract lineage through native Flink interfaces. Supports Flink SQL. Requires Flink 2.0.
• dbt: add support for consuming dbt structured logs for test and build commands . #3362 @MassyB New option for dbt integration now can handle test and build commands too.
• Spark: allow attaching custom facets to RDDExecContext events. #3379 @ssanthanam185 Events emitted from RDDExecutionContext now include custom facets that get loaded as part of InternalHandlerFactory.
• spec: add DatasetTypeDatasetFacet. #3390 @JDarDagran DatasetTypeDatasetFacet allows explicit declaration of type of the resulting dataset.

Changed

• Python: allow adding additionalProperties to Python facets #3391 @JDarDagran Adds with_additonal_properties method that allows to create modified instance of facet with additional properties.
• Spark: allow attaching custom facets to RDDExecContext events. #3379 @ssanthanam185 Events emitted from RDDExecutionContext now include custom facets that get loaded as part of InternalHandlerFactory.
• Spark: SerializedFromObject events aren't filtered for not-delta plans. #3403 @ssanthanam185 Those events shouldn't be filtered outside Databricks/Delta ecosystem.

Fixed

• Spark: fixed ClassLoader handling for OpenLineageExtensionProvider. #3368 @ddebowczyk92 Fixes ClassNotFoundException issue when using the openlineage-spark integration alongside a Spark connector that implements the spark-extension-interfaces due to class loader conflicts.
• SQL: add minimal support for Snowflake LATERAL. #3368 @cisenbe SQL parser won't error on Snowflake's LATERAL keyword.
• dbt: handle errors in parse_assertions #3311 @dsaxton-1password dbt integration won't fail when looking at tests on seeds.
• Spark: fix infinite loop in RDD flatten & perf optimization. #3379 @ssanthanam185 Spark integration now correctly handles complex jobs that have cycles and nested RDD trees.
• Python: FileTransport now correctly attaches json file extension. #3404 @kacpermuda When append=False, the json file extension wasn't properly added before.

Added

• dbt: Consume dbt structured logs and report progress in real time. #3314 @MassyB If --consume-structured-logs flag is set, dbt integration will consume dbt structured logs and report execution progress in real time.
• Java: Add transform transport to allow event modification. #3301 @pawel-big-lebowski New transport type allows to modify the event based on the specified transformer class.
• Java: Parallel event emitting for composite transport. #3305[#3305] @pawel-big-lebowski Emit events in parallel for composite transport. Running in parallel is a default behaviour continueOnFailure set to true. Default value of continueOnFailure got changed from false to true.
• Spark: Collect ScanReport and CommitReport in OpenLineage events when dealing with Iceberg tables. #3256 @pawel-big-lebowski Collects additional Iceberg metrics for datasets read or written through the library. Visit Dataset Metrics docs for more details.
• dbt: add support for duckdb adapter #3280 @mobuchowski Adds support for duckdb adapter for dbt integration.

Changed

• Spark: Add DatasetFactory to support Dataset creation. #3207 @pawel-big-lebowski Adds DatasetFactory to support Dataset creation. This class is used to create Dataset instances for DatasetFactory.

Fixed

• Spark: fix inconsistent dataset naming #3285 @pawel-big-lebowski GCS path now has correctly stripped leading slash

Added

• Dbt: Add support for Column-Level Lineage in dbt integration. #3264 @mayurmadnani Dbt integration now uses SQL parser to add information about collected column-level lineage.
• Spark: Add input and output statistics about datasets read and written. #3240#3263 @pawel-big-lebowski Fix issues related to existing output statistics collection mechanism and fetch input statistics. Output statistics contain now amount of files written, bytes size as well as records written. Input statistics contain bytes size and number of files read, while record count is collected only for DataSourceV2 sources.
• Introduced InputStatisticsInputDatasetFacet #3238 @pawel-big-lebowski
  Extend spec with a new facet InputStatisticsInputDatasetFacet modelled after a similar OutputStatisticsOutputDatasetFacet to contain statistics about input dataset read by a job.

Changed

• Spark: Exclude META-INF/*TransportBuilder from Spark Extension Interfaces #3244 @tnazarew Excludes META-INF/*TransportBuilder to avoid version conflicts
• Spark: enables building input/output facets through DatasetFactory #3207 @pawel-big-lebowski
  Adds extra capabilities into DatasetFactory class, marks some public developers' API methods as deprecated.

Fixed

• dbt: fix compatibility with dbt v1.8 #3228 @NJA010 dbt integration now takes into account modified test_metadata field
• Spark: enabled Delta 3.x version compatibility #3253 @Jorricks
  Take into account modified initialSnapshot name

Added

• Spark: Add Dataproc run facet to include jobType property #3167 @codelixir
  Updates the GCP Dataproc run facet to include jobType property
• Add EnvironmentVariablesRunFacet to core spec #3186 @JDarDagran
  Use EnvironmentVariablesRunFacet in Python client
• Add assertions for format in test events #3221 @JDarDagran
• Spark: Add integration tests for EMR #3142 @arturowczarek
  Spark integration has integration tests for EMR

Changed

• Move Kinesis to separate module, migrate HTTP transport to httpclient5 #3205 @mobuchowski
  Moves Kinesis integration to a separate module and updates HTTP transport to use HttpClient 5.x
• Docs: Upgrade docusaurus to 3.6 #3219 @arturowczarek
• Spark: Limit the Seq size in RddPathUtils::extract() #3148 @codelixir
  Adds flag to limit the logs in RddPathUtils::extract() to avoid OutOfMemoryError for large jobs

Fixed

• Docs: Fix outdated Spark-related docs #3215 @mobuchowski
• Fix docusaurus-mdx-checker errors #3217 @arturowczarek
• [Integration/dbt] Parse dbt source tests #3208 @MassyB
  Fix: Consider dbt sources when looking for test results
• Avoid tests in configurable test #3141 @pawel-leszczynski

Added

• Java: added CompositeTransport #3039 @JDarDagran
  This allows user to specify multiple targets to which OpenLineage events will be emitted.
• Spark extension interfaces: support table extended sources #3062 @Imbruced
  Interfaces are now able to extract lineage from Table interface, not only RelationProvider.
• Java: added GCP Dataplex transport #3043 @ddebowczyk92
  Dataplex transport is now available as a separate Maven package for users that want to send OL events to GCP Dataplex
• Java: added Google Cloud Storage transport #3077 @ddebowczyk92
  GCS transport is now available as a separate Maven package for users that want to send OL events to Google Cloud Storage
• Java: added S3 transport #3129 @arturowczarek
  S3 transport is now available as a separate Maven package for users that want to send OL events to S3
• Java: add option to configure client via environment variables #3094 @JDarDagran
  Specified variables are now autotranslated to configuration values.
• Python: add option to configure client via environment variables #3114 @JDarDagran
  Specified variables are now autotranslated to configuration values.
• Python: add option to add custom headers in HTTP transport #3116 @JDarDagran
  Allows user to add custom headers, for example for auth purposes.
• Column level lineage: add full dataset dependencies #3097 #3098 @arturowczarek
  Now, if datasetLineageEnabled is enabled, and when column level lineage depends on the whole dataset, it does add dataset dependency instead of listing all the column fields in that dataset.
• Java: OpenLineageClient and Transports are now AutoCloseable #3122 @ddebowczyk92
  This prevents a number of issues that might be caused by not closing underlying transports

Fixed

• Python Facet generator does not validate optional arguments #3054 @JDarDagran
  This fixes issue where NominalTimeRunFacet Facet breaks when nominalEndTime is None
• SQL: report only actually used tables from CTEs, rather than all #2962 @Imbruced
  With this change, if SQL specified CTE, but does not use it in final query, the lineage won't be falsely reported
• Fluentd: Enhancing plugin's capabilities #3068 @jonathanlbt1
  This change enhances performance and docs of fluentd proxy plugin.
• SQL: fix parser to point to origin table instead of CTEs #3107 @Imbruced
  For some complex CTEs, parser emitted CTE as a target table instead of original table. This is now fixed.
• Spark: column lineage correctly produces for merge into command #3095 @Imbruced
  Now OL produces CLL correctly for the potential view in the middle.

Added

• SQL: add support for USE statement with different syntaxes #2944 @kacpermuda
  Adjusts our Context so that it can use the new support for this statement in the parser and pass it to a number of queries.
• Spark: add script to build Spark dependencies #3044 @arturowczarek
  Adds a script to rebuild dependencies automatically following releases.
• Website: versionable docs #3007 #3023 @pawel-big-lebowski
  Adds a GitHub action that creates a new Docusaurus version on a tag push, verifiable using the openlineage-site repo. Implements a monorepo approach in a new website directory.

Fixed

• SQL: add support for SingleQuotedString in Identifier() #3035 @kacpermuda
  Single quoted strings were being treated differently than strings with no quotes, double quotes, or backticks.
• SQL: support IDENTIFIER function instead of treating it like table name #2999 @kacpermuda
  Adds support for this identifier in SELECT, MERGE, UPDATE, and DELETE statements. For now, only static identifiers are supported. When a variable is used, this table is removed from lineage to avoid emitting incorrect lineage.
• Spark: fix issue with only one table in inputs from SQL query while reading from JDBC #2918 @Imbruced
  Events created did not contain the correct input table when the query contained multiple tables.
• Spark: fix AWS Glue jobs naming for RDD events #3020 @arturowczarek
  The naming for RDD jobs now uses the same code as SQL and Application events.

Added

• Spec: add GCP Dataproc facet #2987 @tnazarew Registers the Google Cloud Platform Dataproc run facet.

Fixed

• Airflow: update SQL integration code to work with latest sqlparser-rs main #2983 @kacpermuda
  Adjusts the SQL integration after our sqlparser-rs fork has been updated to the latest main.
• Spark: fix AWS Glue jobs naming for SQL events #3001 @arturowczarek
  SQL events now properly use the names of the jobs retrieved from AWS Glue.
• Spark: fix issue with column lineage when using delta merge into command #2986 @Imbruced
  A view instance of a node is now included when gathering data sources for input columns.
• Spark: minor Spark filters refactor #2990 @arturowczarek
  Fixes a number of minor issues.
• Spark: Iceberg tables in AWS Glue have slashes instead of dots in symlinks #2984 @arturowczarek
  They should use slashes and the prefix table/.
• Spark: lineage for Iceberg datasets that are present outside of Spark's catalog is now present #2937 @d-m-h Previously, reading Iceberg datasets outside the configured Spark catalog prevented the datasets from being present in the inputs property of the RunEvent.

Added

• Python: add CompositeTransport #2925 @JDarDagran
  Adds a CompositeTransport that can accept other transport configs to instantiate transports and use them to emit events.
• Spark: compile & test Spark integration on Java 17 #2828 @pawel-big-lebowski
  The Spark integration is always compiled with Java 17, while tests are running on both Java 8 and Java 17 according to the configuration.
• Spark: support preview release of Spark 4.0 #2854 @pawel-big-lebowski
  Includes the Spark 4.0 preview release in the integration tests.
• Spark: add handling for Window #2901 @tnazarew
  Adds handling for Window-type nodes of a logical plan.
• Spark: extract and send events with raw SQL from Spark #2913 @Imbruced
  Adds a parser that traverses QueryExecution to get the SQL query used from the SQL field with a BFS algorithm.
• Spark: support Mongostream source #2887 @Imbruced
  Adds a Mongo streaming visitor and tests.
• Spark: new mechanism for disabling facets #2912 @arturowczarek
  The mechanism makes FacetConfig accept the disabled flag for any facet instead of passing them as a list.
• Spark: support Kinesis source #2906 @Imbruced
  Adds a Kinesis class handler in the streaming source builder.
• Spark: extract DatasetIdentifier from extension LineageNode #2900 @ddebowczyk92
  Adds support for cases in which LogicalRelation has a grandChild node that implements the LineageRelation interface.
• Spark: extract Dataset from underlying BaseRelation #2893 @ddebowczyk92
  DatasetIdentifier is now extracted from the underlying node of LogicalRelation.
• Spark: add descriptions and Marquez UI to Docker Compose file #2889 @jonathanlbt1
  Adds the marquez-web service to docker-compose.yml.

Fixed

• Proxy: bug fixed on error messages descriptions #2880 @jonathanlbt1
  Improves error logging.
• Proxy: update Docker image for Fluentd 1.17 #2877 @jonathanlbt1
  Upgrades the Fluentd version.
• Spark: fix issue with Kafka source when saving with for each batch method #2868 @Imbruced
  Fixes an issue when Spark is in streaming mode and input for Kafka was not present in the event.
• Spark: properly set ARN in namespace for Iceberg Glue symlinks #2943 @arturowczarek
  Makes IcebergHandler support Glue catalog tables and create the symlink using the code from PathUtils.
• Spark: accept any provider for AWS Glue storage format #2917 @arturowczarek
  Makes the AWS Glue ARN generating method accept every format (including Parquet), not only Hive SerDe.
• Spark: return valid JSON for failed logical plan serialization #2892 @arturowczarek
  The LogicalPlanSerializer now returns <failed-to-serialize-logical-plan> for failed serialization instead of an empty string.
• Spark: extract legacy column lineage visitors loader #2883 @arturowczarek
  Refactors CustomCollectorsUtils for improved readability.
• Spark: add Kafka input source when writing in foreach batch mode #2868 @Imbruced
  Fixes a bug keeping Kafka input sources from being produced.
• Spark: extract DatasetIdentifier from SaveIntoDataSourceCommandVisitor options #2934 @ddebowczyk92
  Extracts DatasetIdentifier from command's options instead of relying on p.createRelation(sqlContext, command.options()), which is a heavy operation for JdbcRelationProvider.

Added

• Airflow: add log_url to AirflowRunFacet #2852 @dolfinus
  Adds taskinstance's log_url field to AirflowRunFacet.
• Spark: add handling for Generate #2856 @tnazarew
  Adds handling for Generate-type nodes of a logical plan (e.g., explode operations).
• Java: add DerbyJdbcExtractor #2869 @dolfinus
  Adds JdbcExtractor implementation for Derby database. As this is a file-based DBMS, its Dataset namespace is file and name is an absolute path to a database file.
• Spark: verify bytecode version of the built jar. #2859 @pawel-big-lebowski
  Extends the JarVerifier plugin to ensure all compiled classes have a bytecode version of Java 8 or lower.
• Spark: add Kafka streaming source support #2851 @d-m-h
  Adds support for Kafka streaming sources to Kafka streaming sinks. Inputs and outputs are now included in lineage events.

Fixed

• Airflow: replace datetime.now with airflow.utils.timezone.utcnow #2865 @kacpermuda
  Fixes missing timezone information in task FAIL events.
• Spark: remove shaded dependency in ColumnLevelLineageBuilder #2850 @tnazarew
  Removes the shaded Streams dependency in ColumnLevelLineageBuilder causing a ClassNotFoundException.
• Spark: make Delta dataset symlink consistent with non-Delta tables #2863 @dolfinus
  Makes dataset symlinks for Delta and non-Delta tables consistent.
• Spark: use Table's properties during column-level lineage construction #2855 @ddebowczyk92
  Fixes PlanUtils3 so Dataset identifier information based on a Table's properties is also retrieved during the construction of column-level lineage.
• Spark: extract job name creation to providers #2861 @arturowczarek
  The integration now detects if the spark.app.name was autogenerated by Glue and uses the Glue job name in such cases. Also, each job name provisioning strategy is now extracted to a separate provider.

Added

• Spark: configurable integration test #2755 @pawel-big-lebowski
  Provides command line tool capable of running Spark integration tests that can be created without Java.
• Spark: OpenLineage Spark extension interfaces without runtime dependency hell #2809 #2837 @ddebowczyk92
  New Spark extension interfaces without runtime dependency hell. Includes a test to verify the integration is working properly.
• Spark: support latest versions 3.4.3 and 3.5.1. #2743 @pawel-big-lebowski
  Upgrades CI workflows to run tests against latest Spark versions: 3.4.2 -> 3.4.3 and 3.5.0 -> 3.5.1.
• Spark: add extraction of the masking property in column-level lineage #2789 @tnazarew
  Adds extraction of the masking property during collection of dependencies for ColumnLineageDatasetFacet creation.
• Spark: collect table name from InsertIntoHadoopFsRelationCommand #2794 @dolfinus
  Collects a table name for INSERT INTO command for tables created with USING $fileFormat syntax, like USING orc.
• Spark, Flink: add PostgresJdbcExtractor #2806 @dolfinus Adds the default 5432 port to Postgres namespaces.
• Spark, Flink: add TeradataJdbcExtractor #2826 @dolfinus Converts JDBC URLs like jdbc:teradata/host/DBS_PORT=1024,DATABASE=somedb to datasets with namespace teradata://host:1024 and name somedb.table.
• Spark, Flink: add MySqlJdbcExtractor #2825 @dolfinus Handles different formats of MySQL JDBC URL, and produces datasets with consistent namespaces, like mysql://host:port.
• Spark, Flink: add OracleJdbcExtractor #2824 @dolfinus Handles simple Oracle JDBC URLs, like oracle:thin:@//host:port/serviceName and oracle:thin@host:port:sid, and converts each to a dataset with namespace oracle://host:port and name sid.schema.table or serviceName.schema.table.
• Spark: configurable test with Docker image provided #2822 @pawel-big-lebowski
  Extends the configurable integration test feature to enable getting the Docker image name as a name.
• Spark: Support Iceberg 1.4 on Spark 3.5.1. #2838 @pawel-big-lebowski
  Include Iceberg support for Spark 3.5. Fix column level lineage facet for UNION queries.
• Spec: add example for change in #2756 #2801 @Sheeri Updates the customLineage facet test for the new syntax created in #2756.

Changed

• Spark: fallback to spark.sql.warehouse.dir as table namespace #2767 @dolfinus
  In cases when a metastore is not used, falls back to spark.sql.warehouse.dir or hive.metastore.warehouse.dir as table namespace, instead of duplicating the table's location.

Fixed

• Java: handle dashes in hostname for JdbcExtractors #2830 @dolfinus
  Proper handling of dashes in JDBC URL hosts.
• Spark: fix Glue symlinks formatting bug #2807 @Akash2351 Fixes Glue symlinks with config parsing for Glue catalogid.
• Spark, Flink: fix DBFS namespace format #2800 @dolfinus Fixes the DBFS namespace format.
• Spark: fix Glue naming format #2766 @dolfinus
  Changes the AWS Glue namespace to match Glue ARN documentation.
• Spark: fix Iceberg dataset location #2797 @dolfinus
  Fixes Iceberg dataset namespace: instead of file:/some/path/database.table uses file:/some/path/database/table. For dataset TABLE symlink, uses warehouse location instead of database location.
• Spark: fix NPE and incorrect comment #2827 @pawel-big-lebowski Fixes an error caused by a recent upgrade of Spark versions that did not break existing tests.
• Spark: convert scheme and authority to lowercase in JdbcLocation #2831 @dolfinus
  Converts valid JDBC URL scheme and authority to lowercase, leaving intact instance/database name, as different databases have different default case and case-sensitivity rules.

Added

• Java: dataset namespace resolver feature #2720 @pawel-big-lebowski
  Adds a dataset namespace resolving mechanism that resolves dataset namespaces based on the resolvers configured. The core mechanism is implemented in openlineage-java and can be used within the Flink and Spark integrations.
• Spark: add transformation extraction #2758 @tnazarew
  Adds a transformation type extraction mechanism.
• Spark: add GCP run and job facets #2643 @codelixir
  Adds GCPRunFacetBuilder and GCPJobFacetBuilder to report additional facets when running on Google Cloud Platform.
• Spark: improve namespace format for SQLServer #2773 @dolfinus
  Improves the namespace format for SQLServer.
• Spark: verify jar content after build #2698 @pawel-big-lebowski
  Adds a tool to verify shadowJar content and prevent reported issues. These are hard to prevent currently and require manual verification of manually unpacked jar content.
• Spec: add transformation type info #2756 @tnazarew
  Adds information about the transformation type in ColumnLineageDatasetFacet. transformationType and transformationDescription are marked as deprecated.
• Spec: implementing facet registry (following #2161) #2729 @harels
  Introduces the foundations of the new facet Registry into the repo.
• Spec: register GCP common job facet #2740 @ngorchakova
  Registers the GCP job facet that contains common attributes that will improve the way lineage is parsed and displayed by the GCP platform. Based on the proposal, GCP Lineage would like to define facets that are expected from integrations. The list of support facets is not final and will be extended further by next PR.

Removed

• Java: remove deprecated localServerId option from Kafka config #2738 @dolfinus
  Removes localServerId from Kafka config, deprecated since 1.13.0.
• Java: remove deprecated Transport.emit(String) #2737 @dolfinus
  Removes Transport.emit(String) support, deprecated since 1.13.0.
• Spark: remove spark-interfaces-scala module #2781 @ddebowczyk92
  Replaces the existing spark-interfaces-scala interfaces with new ones decoupled from the Scala binary version. Allows for improved integration in environments where one cannot guarantee the same version of openlineage-java.

Changed

• Spark: add log info when emitting lineage from Spark (following #2650) #2769 @algorithmy1
  Enhances logging.

Fixed

• Flink: use namespace.name as Avro complex field type #2763 @dolfinus
  namespace.name is now used as Avro "type" of complex fields (record, enum, fixed).
• Java: repair empty dataset name #2776 @kacpermuda
  The dataset name should not be empty.
• Spark: fix events emitted for drop table for Spark 3.4 and above #2745 @pawel-big-lebowski @savannavalgi
  Includes dataset being dropped within the event, as it used to be prior to Spark 3.4.
• Spark, Flink: fix S3 dataset names #2782 @dolfinus
  Drops the leading slash from the object storage dataset name. Converts s3a:// and s3n:// schemes to s3://.
• Spark: fix Hive metastore namespace #2761 @dolfinus
  Fixes the dataset namespace for cases when the Hive metastore URL is set using $SPARK_CONF_DIR/hive-site.xml.
• Spark: fix NPE in column-level lineage #2749 @pawel-big-lebowski
  The Spark agent now checks to determine if cur.getDependencies() is not null before adding dependencies.
• Spark: refactor OpenLineageRunEventBuilder #2754 @pawel-big-lebowski
  Adds a separate class containing all the input arguments to call OpenLineageRunEventBuilder::buildRun.
• Spark: fix historyUrl format #2741 @dolfinus
  Fixes the historyUrl format in spark_applicationDetails.
• SQL: allow self-recursive aliases #2753 @mobuchowski
  Expressions like select * from test_orders as test_orders are now parsed properly.

Added

• Spark: add jobType facet to Spark application events #2719 @dolfinus Adds jobType facet to runEvents emitted by SparkListenerApplicationStart.

Fixed

• dbt: fix swapped namespace and name in dbt integration #2735 @JDarDagran Fixes variable names.
• Python: override debug level #2727 @mobuchowski Removes debug-level logging of HTTP requests.

Added

• Flink: handle Iceberg tables with nested and complex field types #2706 @dolfinus
  Creates SchemaDatasetFacet with nested fields for Iceberg tables with list, map and struct columns.
• Flink: handle Avro schema with nested and complex field types #2711 @dolfinus
  Creates SchemaDatasetFacet with nested fields for Avro schemas with complex types (union, record, map, array, fixed).
• Spark: add facets to Spark application events #2677 @dolfinus
  Adds support for Spark application start and stop events in the ExecutionContext interface.
• Spark: add nested fields to SchemaDatasetFieldsFacet #2689 @dolfinus
  Adds nested Spark Dataframe fields support to SchemaDatasetFieldsFacet. Also include field comment as description.
• Spark: add SparkApplicationDetailsFacet #2688 @dolfinus
  Adds SparkApplicationDetailsFacet to runEvents emitted on Spark application start.

Removed

• Airflow: remove Airflow < 2.3.0 support #2710 @kacpermuda
  Removes Airflow < 2.3.0 support.
• Integration: use v2 Python facets #2693 @JDarDagran
  Migrates integrations from removed v1 facets to v2 Python facets.

Fixed

• Spark: improve job suffix assigning mechanism #2665 @pawel-big-lebowski
  For some catalog handlers, the mechanism was creating different dataset identifiers on START and COMPLETE depending on whether a dataset was created or not. This improves the mechanism to assign a deterministic job suffix based on the output dataset at the moment of a start event. Note: this may change job names in some scenarios.
• Airflow: fix empty dataset name for AthenaExtractor #2700 @kacpermuda
  The dataset name should not be empty when passing only a bucket as S3 output in Athena.
• Flink: fix SchemaDatasetFacet for Protobuf repeated primitive types #2685 @dolfinus
  Fixes issues with the Protobuf schema converter.
• Python: clean up Python client code, add logging. #2653 @kacpermuda
  Cleans up client code, refactors logging in all Python modules.
• SQL: catch TokenizerErrors, PanicException #2703 @mobuchowski
  The SQL parser now catches and handles these errors.
• Python: suppress warning on importing v1 module in init.py. #2713 @JDarDagran
  Suppresses the deprecation warning when v1 facets are used.
• Integration/Java/Python: use UUIDv7 instead of UUIDv4 #2686 #2687 @dolfinus
  Uses UUIDv7 instead of UUIDv4 for runEvents. The new UUID version produces monotonically increasing values, which leads to more performant queries on the OL consumer side. Note: UUID version is an implementation detail and can be changed in the future.

Added

• Common/dbt: add DREMIO to supported dbt profile types #2674 @surisimran
  Adds support for dbt-dremio, resolving #2668.
• Flink: support Protobuf format for sources and sinks #2482 @pawel-big-lebowski
  Adds schema extraction from Protobuf classes. Includes support for nested object types, array type, map type, oneOf and any.
• Java: add facet conversion test #2663 @julienledem
  Adds a simple test that shows how to deserialize a facet in the server model.
• Spark: job type facet to distinguish RDD jobs from Spark SQL jobs #2652 @pawel-big-lebowski
  Sets the jobType property of JobTypeJobFacet to either SQL_JOB or RDD_JOB.
• Spark: add Glue symlink if reading from Glue catalog table #2646 @mobuchowski
  The dataset symlink now points to the Glue catalog table name if the Glue catalog table is used.
• Spark: add spark_jobDetails facet #2662 @dolfinus Adds a SparkJobDetailsFacet, capturing information about Spark application jobs -- e.g. jobId, jobDescription, jobGroup, jobCallSite. This allows for tracking an OpenLineage RunEvent with a specific Spark job in SparkUI.

Removed

• Airflow: drop old ParentRunFacet key #2660 @dolfinus Changes the integration to use the parent key for ParentFacet, dropping the outdated parentRun.
• Spark: drop SparkVersionFacet #2659 @dolfinus Drops the SparkVersion facet, deprecated since 1.2.0 and planned for removal since 1.4.0.
• Python: allow relative paths in URI formats for Python facets #2679 @JDarDagran
  Removes a URI validator that checked if scheme and netloc were present, allowing relative paths in URI formats for Python facets.

Changed

• GreatExpectations: rename ParentRunFacet key #2661 @dolfinus
  The OpenLineage spec defined the ParentRunFacet with the property name parent but the Great Expectations integration created a lineage event with parentRun. This renames ParentRunFacet key from parentRun to parent. For backwards compatibility, keep the old name.

Fixed

• dbt: support a less ambiguous logic to generate job names #2658 @blacklight Includes profile and models in the dbt job name to make it more unique.
• Spark: update to use org.apache.commons.lang3 instead of org.apache.commons.lang #2676 @harels
  Updates Apache Commons Lang to the latest version. We were mixing two versions, and the old one was not present in many places.

Added

• Java: allow timeout for circuit breakers #2609 @pawel-big-lebowski
  Extends the circuit breaker mechanism to contain a global timeout that stops running OpenLineage integration code when a specified amount of time has elapsed.
• Java: handle DataSetEvent and JobEvent in Transport.emit #2611 @dolfinus
  Adds overloads Transport.emit(OpenLineage.DatasetEvent) and Transport.emit(OpenLineage.JobEvent), reusing the implementation of Transport.emit(OpenLineage.RunEvent). Please note: Transport.emit(String) is now deprecated and will be removed in 1.16.0.
• Java/Python: add GZIP compression to HttpTransport #2603 #2604 @dolfinus
  Adds a compression option to HttpTransport config in the Java and Python clients, with gzip implementation.
• Java/Python/Proxy: properly set Kafka message key #2571 #2597 #2598 @dolfinus
  Adds a new messageKey option to KafkaTransport config in the Python and Java clients, as well as the Proxy. This option replaces the localServerId option, which is now deprecated. Default value is generated using the run id (for RunEvent), job name (for JobEvent) or dataset name (for DatasetEvent). This value is used by the Kafka producer to distribute messages along topic partitions, instead of sending all the events to the same partition. This allows for full utilization of Kafka performance advantages.
• Flink: add support for Micrometer metrics #2633 @mobuchowski Adds a mechanism for forwarding metrics to any Micrometer-compatible implementation for Flink as has been implemented for Spark. Included: MeterRegistry, CompositeMeterRegistry, SimpleMeterRegistry, and MicrometerProvider.
• Python: generate Python facets from JSON schemas #2520 @JDarDagran
  Objects specified with JSON Schema needed to be manually developed and checked in Python, leading to many discrepancies, including wrong schema URLs. This adds a datamodel-code-generator for parsing JSON Schema and generating Pydantic or dataclasses classes, etc. In order to use attrs (a more modern version of dataclasses) and overcome some limitations of the tool, a number of steps have been added in order to customize code to meet OpenLineage requirements. Included: updated references to the latest base JSON Schema spec for all child facets. Please note: newly generated code creates a v2 interface that will be implemented in existing integrations in a future release. The v2 interface introduces some breaking changes: facets are put into separate modules per JSON Schema spec file, some names are changed, and several classes are now kw_only.
• Spark/Flink/Java: support YAML config files together with SparkConf/FlinkConf #2583 @pawel-big-lebowski
  Creates a SparkOpenlineageConfig and FlinkOpenlineageConfig for a more uniform configuration experience for the user. Renames OpenLineageYaml to OpenLineageConfig and modifies the code to use only OpenLineageConfig classes. Includes a doc update to mention that both ways can be used interchangeably and final documentation will merge all values provided.
• Spark: add custom token provider support #2613 @tnazarew Adds a TokenProviderTypeIdResolver to handle both FQCN and (for backward compatibility) api_key types in spark.openlineage.transport.auth.type.
• Spark/Flink: support YAML config files together with SparkConf & FlinkConf approaches #2583 @pawel-big-lebowski
  Adds support for config entries being provided by both YAML file and integration-specific configuration (SparkConf/FlinkConf). Allows each integration to have its own config entries.
• Spark/Flink: job ownership facet #2533 @pawel-big-lebowski Enables configuration entries specifying ownership of the job that will result in an OwnershipJobFacet being attached to job facets.

Changed

• Java: sync Kinesis partitionKey format with Kafka implementation #2620 @dolfinus Changes the format of Kinesis partitionKey from {jobNamespace}:{jobName} to run:{jobNamespace}/{jobName} to match the Kafka transport implementation.

Fixed

• Python: make load_config return an empty dict instead of None when file empty #2596 @kacpermuda utils.load_config() now returns an empty dict instead of None in the case of an empty file to prevent an OpenLineageClient crash.
• Java: render lombok-generated methods in javadoc #2614 @dolfinus Fixes rendering of javadoc for methods generated by lombok annotations by adding a delombok step.
• Spark/Snowflake: parse NPE when query option is used and table is empty #2599 @mobuchowski
  Fixes NPE when using query option when reading from Snowflake.

Added

• Airflow: add lineage_job_namespace and lineage_job_name macros #2582 @dolfinus
  Adds new Airflow macros lineage_job_namespace(), lineage_job_name(task) that return an Airflow namespace and Airflow job name, respectively.
• Spec: Allow nested struct fields in SchemaDatasetFacet #2548 @dolfinus
  Adds nested fields support to SchemaDatasetFacet.

Fixed

• Spark: fix PMD for test #2588 @pawel-big-lebowski
  Clears pmdTestScala212 of warnings that clutter the logs.
• Dbt: propagate the dbt return code also when no OpenLineage events are emitted #2591 @blacklight
  dbt-ol now propagates the exit code of the underlying dbt process even if no lineage events are emitted.
• Java: make sure string isn't empty to prevent going out of bounds #2585 @harels
  String lookup was not accounting for empty strings and causing a java.lang.StringIndexOutOfBoundsException.
• Spark: use HashSet in column-level lineage instead of iterating through LinkedList #2584 @mobuchowski
  Takes advantage of performance gains available from using HashSet for collection.
• Python: fix missing pkg_resources module on Python 3.12 #2572 @dolfinus
  Removes pkg_resources dependency and replaces it with the packaging lib.
• Airflow: fix format returned by airflow.macros.lineage_parent_id #2578 @blacklight
  Fixes the run format returned by the lineage_parent_id Airflow macro and simplifies the format of the lineage_parent_id and lineage_run_id macros.
• Dagster: limit Dagster version to 1.6.9 #2579 @JDarDagran Adds an upper limit on supported versions of Dagster as the integration is no longer actively maintained and recent releases have introduced breaking changes.

Added

• Common: add support for SCRIPT-type jobs in BigQuery #2564 @kacpermuda In the case of SCRIPT-type jobs in BigQuery, no lineage was being extracted because the SCRIPT job had no lineage information - it only spawned child jobs that had that information. With this change, the integration extracts lineage information from child jobs when dealing with SCRIPT-type jobs.
• Spark: support for built-in lineage extraction #2272 @pawel-big-lebowski This PR adds a spark-interfaces-scala package that allows lineage extraction to be implemented within Spark extensions (Iceberg, Delta, GCS, etc.). The Openlineage integration, when traversing the query plan, verifies if nodes implement defined interfaces. If so, interface methods are used to extract lineage. Refer to the README for more details.
• Spark/Java: add support for Micrometer metrics #2496 @mobuchowski
  Adds a mechanism for forwarding metrics to any Micrometer-compatible implementation. Included: MeterRegistryyFactory, MicrometerProvider, StatsDMetricsBuilder, metrics config in OpenLineage config, and a Java client implementation.
• Spark: add support for telemetry mechanism #2528 @mobuchowski
  Adds timers, counters and additional instrumentation in order to implement Micrometer metrics collection.
• Spark: support query option on table read #2556 @mobuchowski Adds support for the Spark-BigQuery connector's query input option, which executes a query directly on BigQuery, storing the result in an intermediate dataset, bypassing Spark's computation layer. Due to this, the lineage is retrieved using the SQL parser, similarly to JDBCRelation.
• Spark: change SparkPropertyFacetBuilder to support recording Spark runtime #2523 @Ruihua98
  Modifies SparkPropertyFacetBuilder to capture the RuntimeConfig of the Spark session because the existing SparkPropertyFacet can only capture the static config of the Spark context. This facet will be added in both RDD-related and SQL-related runs.
• Spec: add fileCount to dataset stat facets #2562 @dolfinus
  Adds a fileCount field to DataQualityMetricsInputDatasetFacet and OutputStatisticsOutputDatasetFacet specification.

Fixed

• dbt: dbt-ol should transparently exit with the same exit code as the child dbt process #2560 @blacklight
  Makes dbt-ol transparently exit with the same exit code as the child dbt process.
• Flink: disable module metadata generation #2531 @HuangZhenQiu
  Disables the module metadata generation for Flink to fix the problem of having gradle dependencies to submodules within openlineage-flink.jar.
• Flink: fixes to version 1.19 #2507 @pawel-big-lebowski Fixes the class not found issue when checking for Cassandra classes. Also fixes the Maven pom dependency on subprojects.
• Python: small improvements to .emit() method logging & annotations #2539 @dolfinus
  Updates OpenLineage.emit debug messages and annotations.
• SQL: show error message when OpenLineageSql cannot find native library #2547 @dolfinus
  When the OpenLineageSql class could not load a native library, if returned None for all operations. But because the error message was suppressed, the user could not determine the reason.
• SQL: update code to conform to upstream sqlparser-rs changes #2510 @mobuchowski
  Includes tests and cosmetic improvements.
• Spark: fix access to active Spark session #2535 @pawel-big-lebowski
  Changes behavior so IllegalStateException is always caught when accessing SparkSession.
• Spark: fix Databricks environment #2537 @pawel-big-lebowski
  Fixes the ClassNotFoundError occurring on Databricks runtime and extends the integration test to verify DatabricksEnvironmentFacet.
• Spark: fixed memory leak in JobMetricsHolder #2565 @d-m-h The JobMetricsHolder#cleanUp(int) method now correctly purges unneeded state from both maps.
• Spark: fixed memory leak in UnknownEntryFacetListener #2557 @pawel-big-lebowski Prevents storing the state when a facet is disabled, purging the state after populating run facets.
• Spark: fix parsing JDBCOptions(table=...) containing subquery #2546 @dolfinus
  Prevents openlineage-spark from producing datasets with names like database.(select * from table) for JDBC sources.
• Spark/Snowflake: support query option via SQL parser #2563 @mobuchowski
  When a Snowflake job is bypassing Spark's computation layer, now the SQL parser will be used to get the lineage.
• Spark: always catch IllegalStateException when accessing SparkSession #2535 @pawel-big-lebowski IllegalStateException was not being caught.

Added

• Dagster: add new provider for version 1.6.10 #2518 @JDarDagran Adds the new provider required by the latest version of Dagster.
• Flink: support lineage for a hybrid source #2491 @HuangZhenQiu Adds support for hybrid source lineage for users of Kafka and Iceberg sources in backfill usecases.
• Flink: bump Flink JDBC connector version #2472 @HuangZhenQiu Bumps the Flink JDBC connector version to 3.1.2-1.18 for Flink 1.18.
• Java: add a OpenLineageClientUtils#loadOpenLineageJson(InputStream) and change OpenLineageClientUtils#loadOpenLineageYaml(InputStream) methods #2490 @d-m-h This improves the explicitness of the methods. Previously, loadOpenLineageYaml(InputStream) wanted the InputStream to contain bytes that represented JSON.
• Java: add info from the HTTP response to the client exception #2486 @davidjgoss Adds the status code and body as properties on the thrown exception when a non-success response is encountered in the HTTP transport.
• Python: add support for MSK IAM authentication with a new transport #2478 @mattiabertorello
  Eases publication of events to MSK with IAM authentication.

Removed

• Airflow: remove redundant information from facets #2524 @kacpermuda Refines the operator's attribute inclusion logic in facets to include only those known to be important or compact, ensuring that custom operator attributes with substantial data do not inflate the event size.

Fixed

• Airflow: proceed without rendering templates if task_instance copy fails #2492 @kacpermuda Airflow will now proceed without rendering templates if task_instance copy fails in listener.on_task_instance_running.
• Spark: fix the HttpTransport timeout #2475 @pawel-big-lebowski The existing timeout config parameter is ambiguous: implementation treats the value as double in seconds, although the documentation claims it's milliseconds. A new config param timeoutInMillis has been added. the Existing timeout has been removed from docs and will be deprecated in 1.13.
• Spark: prevent NPE if the context is null #2515 @pawel-big-lebowski Adds a check for a null context before executing end(jobEnd).
• Flink: fix class not found issue for Cassandra #2507 @pawel-big-lebowski Fixes the class not found issue when checking for Cassandra classes. Also fixes the Maven POM dependency on subprojects.
• Flink: refine the JDBC table name #2512 @HuangZhenQiu Enables the JDBC table name with a schema prefix.
• Flink: fix JDBC dataset naming #2508 @pawel-big-lebowski For JDBC, the Flink integration is not adjusted to the Openlineage naming convention. There is code that extracts the dataset namespace/name from the JDBC connection url, but it's in the Spark integration. As a solution, this code has to be extracted into the Java client and reused by the Spark and Flink integrations.
• Flink: fix failure due to missing Cassandra classes #2507 @pawel-big-lebowski Flink is failing when no Cassandra classes are present on the class path. This is happening because of CassandraUtils class which has a static hasClasses method, but it imports Cassandra-related classes in the header. Also, the Flink subproject contains an unnecessary maven-publish plugin.
• Flink: fix release runtime dependencies #2504 @HuangZhenQiu The shadow jar of Flink is not minimized, so some internal jars are listed as runtime dependences. This removes them from the final pom.xml file in the Flink module.
• Spec: improve Cassandra lineage metadata #2479 @HuangZhenQiu Following the namespace definition, we should use cassandra://host:port.