Fixed

• Python: re-add missing version variables in top of releaseable modules #4135 @mobuchowski Fixes breaking change in version 1.40.0.

Added

• Spec: standardize batch API endpoint #4109 @jakub-moravec Add a standardized batch API endpoint to OpenLineage specification for handling multiple events in a single request.
• Spec: Add ordinal position to SchemaDatasetFacet #4116 @mobuchowski Add ordinal_position field to track the position of fields in schema (1-indexed).
• Spec: Add JobDependenciesRunFacet #4112 @kacpermuda Introduce JobDependenciesRunFacet to track dependencies between jobs.
• Spec: Add support for temporary datasets #4103 @jakub-moravec Add support for temporary datasets to enable job-to-job lineage tracking.
• Spark: Add fallback for BigQuery project ID configuration #4075 @luke-hoffman1 Add fallback configuration for BigQuery project ID in Metastore integration.
• Spark: Add COALESCE transformation support #4123 @kacpermuda Include examples in Python generated classes for better documentation.
• Java: Add support for jTDS JDBC URL format #4077 @dolfinus Add support for parsing jTDS JDBC URL format in Java client.
• Hive: Add ParentRunFacet #4066 @tnazarew Add ParentRunFacet to Hive integration for tracking parent-child run relationships.
• Hive: Add LOAD and IMPORT handling #4097 @tnazarew Add support for tracking LOAD and IMPORT operations in Hive.
• Hive: Add EXPORT handling #4085 @tnazarew Add support for tracking EXPORT operations in Hive.
• Hive: Add START event emission #4079 @tnazarew Add START event emission support to Hive integration.

Fixed

• Spark: Fix dataset facet builders for inputs #4121 @usamakunwar Fix Spark dataset facet builders for input datasets.
• Spark: Fix job name trimming #4114 @kchledowski Fix job name trimming logic in Spark integration.
• Spark: Fix putAll on immutable maps #4113 @pawel-big-lebowski Fix putAll operation failing on immutable maps.
• Spark: Fix RDD job handling #4108 @pawel-big-lebowski Fix multiple issues with RDD job handling in Spark.
• Spark: Fix JDBC dbtable parsing #4102 @kchledowski Fix JDBC dbtable parsing to support any FROM clauses.
• Spark: Fix Databricks setup #4083 @pawel-big-lebowski Fix Spark connector configuration for Databricks environments.
• Spark: Catch NoClassDefFoundError #4099 @mobuchowski Catch NoClassDefFoundError when buggy implementations exist on classpath.
• Spark: Fix Snowflake identifier parsing #4104 @mobuchowski Fix Snowflake identifier parsing to handle quoted identifiers correctly.
• Spark: Fix Snowflake account name handling #4105 @mobuchowski Strip quotes from Snowflake account names for proper handling.
• Spec: Fix facet property names #4092 @fm100 Fix facet property names from snake_case to camelCase for consistency.
• Python: Fix facet generator after UV migration #4111 @kacpermuda Fix Python client facet generator after moving to UV build system.
• Python: Fix retry config merge #4093 @antonlin1 Fix retry configuration default merge with user-defined config in HTTP transports.
• Java: Fix CVE in commons-lang3 #4084 @mandalbalmukund Upgrade commons-lang3 version to fix CVE security vulnerability.
• Hive: Generate same runId for START and STOP events #4126 @dolfinus Ensure START and STOP events share the same runId in Hive integration.

Added

• Spark: Normalize dataset names with configurable trimmers #3996 @pawel-big-lebowski Add configurable dataset name normalization with support for date patterns, key-value pairs, and S3 location detection to enable proper dataset subsetting.
• Spark: Add missing facets in inputs for Databricks Unity Catalog #4057 @kchledowski Add missing input symlink facets for Databricks Unity Catalog tables.

Changed

• Spark: Refactor tests for dependency collector #4058 @kchledowski Refactor column-level lineage dependency collector tests for better organization and maintainability.

Fixed

• Spec: Fix typo in iceberg commit report facet spec file #4069 @fm100 Fix typo in IcebergCommitReportOutputDatasetFacet property name.
• Spark: Fix dataset trimming for CLL inputs #4061 @pawel-big-lebowski Fix dataset name trimming for column-level lineage inputs.
• Python: Remove numpy import #4062 @kacpermuda Remove unnecessary numpy import from Python client.

Removed

• Dagster: Remove Dagster integration #3844 @kacpermuda Remove Dagster integration from the repository.

Added

• Spec: Add subset dataset facets to spec #4008 @pawel-big-lebowski Add subset dataset facets to OpenLineage specification for representing dataset relationships.
• Spec: Add DatasetQualityMetricsDatasetFacet #3978 @heron-- Allow attaching dataset quality information outside of InputDatasetFacet.
• Spark: Add support for microbatch source write #4018 @tnazarew Add support for Spark structured streaming microbatch source write operations.
• Spark: Add catalog properties to catalog facet #4016 @ddebowczyk92 Add catalog properties support to Spark integration for better catalog metadata tracking.
• Spark: Add GCP project ID and location to BigQuery Metastore catalog properties #4039 @ddebowczyk92 Enhance BigQuery integration with GCP project ID and location in catalog properties.
• Spark: Add support for COALESCE transformation #3972 @kchledowski Add support for tracking COALESCE transformations in Spark jobs.
• Spark: Add catalog facet when using vanilla Hive tables #3982 @ddebowczyk92 Add catalog facet support for vanilla Hive table operations.
• Spark: Make output statistics available within complete event #4013 @pawel-big-lebowski Output statistics now available in complete events for better observability.
• Spark: Add output stats for RDD jobs #3977 @pawel-big-lebowski Add output statistics tracking for Spark RDD-based jobs.
• Java: Add equals and hashcode methods into generated classes #4050 @pawel-big-lebowski Improve generated model classes with proper equals and hashcode implementations.
• dbt: Capture dbt tags #4022 @mobuchowski Add support for capturing dbt tags in OpenLineage events.
• dbt: Add dbt Cloud account ID to DbtRunRunFacet #4017 @mobuchowski Add dbt Cloud account ID tracking to dbt run facets.
• dbt: Update DbtRunRunFacet to add more useful information #3987 @mobuchowski Enhance DbtRunRunFacet with additional metadata for better observability.
• Python: Add GCP Lineage transport #4006 @ddebowczyk92 Add native Google Cloud Platform Lineage transport for Python client.
• Python: Add fsspec support for FileTransport #3983 @JDarDagran Add fsspec filesystem support to FileTransport for broader filesystem compatibility.
• Python: Add default tags with OL client version #3980 @kacpermuda Automatically add OpenLineage client version as default tag in events.
• Airflow: Add GCP Composer facets #3986 @gabrysiaolsz Add GCP Cloud Composer environment metadata facets to Airflow integration.

Changed

• dbt: Use alias when naming datasets #4055 @mobuchowski Use dbt model aliases when generating dataset names for more accurate lineage.
• Spark: Serialize event to JSON for logging #4029 @EugeneYushin Serialize OpenLineage events to JSON format for improved debug logging.
• Spark: Respect overridden appName in EventEmitter #4030 @EugeneYushin Properly respect user-overridden application names in event emission.
• Spark: Refactor CLL ExpressionDependencyCollector #4003 @kchledowski Refactor column-level lineage expression dependency collector for better maintainability.
• Spark: Improve logging in IcebergInputStatisticsInputDatasetFacetBuilder #3994 @JDarDagran Enhance logging for Iceberg input statistics collection.
• Spark: Limit external getFileStatus calls when dealing with lots of S3 objects #3985 @pawel-big-lebowski Optimize S3 operations by limiting external getFileStatus calls for large object sets.
• Java/Spark/Hive: Move TransformationInfo to Java client to reuse across integrations #3964 @kchledowski Refactor TransformationInfo into shared Java client for cross-integration reuse.
• Python: Improve logging in AsyncHttpTransport #4026 @dolfinus Enhance logging capabilities in asynchronous HTTP transport.
• Python: Allow type aliases #4000 @JDarDagran Support Python type aliases in client code generation.
• Python: Fix classes generation for almost identical classes #3997 @JDarDagran Improve code generation to properly handle nearly identical class definitions.
• Python: Raise errors if custom token provider cannot be loaded #4014 @dolfinus Fail fast with clear errors when custom token providers fail to load.
• Python: Don't silence import errors in DefaultTransportFactory #4015 @dolfinus Improve error visibility by not silencing import errors in transport factory.
• Python: Import from facet_v2 and event_v2 instead of generated modules #3968 @kacpermuda Update import paths to use versioned facet and event modules.
• Java: Refactor ExecutorService management in OpenLineageClientUtils #4012 @JDarDagran Improve thread pool management in Java client utilities.
• CI: Replace pre-commit with prek across CI and documentation #3965 @JDarDagran Migrate from pre-commit to prek for pre-commit hook management.

Fixed

• Spark: Fix false Hive Glue detection #4053 @jsjasonseba Fix incorrect Glue catalog detection due to always attempting ARN resolution.
• Spark: Fix CLL on hiveless runtimes #4052 @kchledowski Fix column-level lineage failures on Spark runtimes without spark-hive package.
• Spark: Fix missing inputs and CLL on some table creation commands #4031 @kchledowski Fix missing input datasets and column-level lineage for CreateDataSourceTableAsSelect and CreateHiveTableAsSelect commands.
• Spark: Rely on BQ bucket info inside BigQueryIntermediateJobFilter #4044 @EugeneYushin Fix BigQuery intermediate job filtering by using bucket configuration.
• Spark: Fix for TypeNotPresentException/RefreshTableCommand errors in Spark 3.0.2 #4002 @MaciejGajewski Add additional exception handling for TypeNotPresentException in Spark 3.0.2.
• Python: Fix license field in pyproject.toml when using build module #4034 @JDarDagran Correct license field specification in Python package metadata.
• Python: Accept both apikey and api_key in token provider #4045 @kacpermuda Support both naming conventions for API key configuration parameter.
• Java: Fix empty sources jar generation #4037 @EugeneYushin Fix build issue causing empty sources JAR files to be generated.

Added

• Python: Add Datadog transport with configurable async routing #3950 @mobuchowski
  Add Datadog transport with intelligent routing between sync/async transports based on configurable rules. Supports wildcard matching and provides seamless integration with Datadog's observability platform.
• Spark: Implement support for WriteDelta, WriteIcebergDelta logical plan nodes #3860 @orthoxerox
  Add support for WriteDelta and WriteIcebergDelta logical plan nodes in Spark integration.
• dbt: Add option to override dbt job name #3933 @mobuchowski
  Add configuration option to override dbt job names in OpenLineage events.
• Java: Add Jackson Blackbird module for JSON performance optimization #3923 @kyungryun
  Improve JSON serialization performance with Jackson Blackbird module.

Changed

• Spark: Remove Spark 2 support #3904 @pawel-big-lebowski
  Drop support for Spark 2.x versions. Minimum supported version is now Spark 3.x.
• Python: Change gzip compression level in HTTP transport #3956 @dolfinus
  Optimize HTTP transport performance by adjusting gzip compression level.
• Spark: Add support for Spark 4 in streaming tests #3925 @SalvadorRomo
  Extend streaming integration tests to support Spark 4.0.

Fixed

• Spark: Improve performance of column level lineage #3946 @pawel-big-lebowski
  Limit memory consumption, provide limits for the amount of dependencies processed (1M) and input fields returned in the facet (100K). Turns on dataset lineage by default.
• Spark: Add schema size limit for column level lineage processing #3949 @ddebowczyk92
  Add limits to prevent performance issues with large schemas in column-level lineage processing.
• Spark: Fix context factory for Spark 4 #3934 @pawel-big-lebowski
  Fix context factory implementation for Spark 4.0 compatibility.
• Spark: Fix LogicalRelation constructor compatibility for Spark 4 #3930 @yunchipang
  Fix LogicalRelation constructor to maintain compatibility with Spark 4.0.
• Spark: Fix vendors parsing in SparkOpenLineageConfig #3947 @ddebowczyk92
  Fix parsing of vendor configurations in Spark OpenLineage configuration.
• dbt: Use correct namespace for dbt externalQuery facet #3953 @jroachgolf84
  Fix namespace handling in dbt external query facets.
• Python: Fix tags configuration #3943 @JDarDagran
  Fix configuration handling for user-supplied tags in Python client.

Added

• Spark: support Delta 4.0 and cover it with tests on Spark 4.0. #3877 @pawel-big-lebowski
  Fix failing tests for Spark 4.0. Make delta integration tests pass with Delta 4.0 on Spark 4.
• Spark: Add memory info to debug facet. #3914 @pawel-big-lebowski
  Extend DebugFacet with additional information on Spark's driver memory configuration and current memory usage.
• Spark: Add new AlterTableCommandDatasetBuilder for Spark 4.0. #3921 @pawel-big-lebowski
  Add support for AlterTableCommand dataset building in Spark 4.0.
• dbt: Add query IDs for dbt. #3890 @jroachgolf84
  Add query ID tracking to dbt integration.
• dbt: Add query ID capture in structured logs. #3918 @mobuchowski
  Capture query IDs from dbt structured logs for better traceability.
• Python: Formalize dataset naming for Python client. #3816 @ddebowczyk92
  Formalize dataset naming conventions in Python client implementation.

Changed

• Spark: bump minor versions 3.4.3 -> 3.4.4, 3.5.4 -> 3.5.6. #3907 @pawel-big-lebowski
  Bump tested Spark versions.
• Spark: Close OpenLineageClient in onApplicationEnd. #3851 @dolfinus
  Ensure proper cleanup of OpenLineageClient when Spark application ends.
• Python: Do not use f-strings with logging module. #3895 @dolfinus
  Replace f-string usage in logging calls with proper logging formatting.
• Python: Update protobuf version to be compatible with newer libraries. #3899 @Shadi
  Update protobuf dependency to maintain compatibility with newer library versions.
• Website: Documentation for compatibility tests. #3869 @mobuchowski
  Add documentation explaining compatibility testing processes.

Fixed

• Spark: make visitors stateless - avoid memory leak. #3902 @pawel-big-lebowski
  Merge SqlExecutionRDDVisitor and LogicalRDDVisitor classes to avoid memory leak.
• Spark: refactor iceberg handler. #3909 @pawel-big-lebowski
  Refactor Iceberg handler implementation for better maintainability.
• Spark: retry exception on empty row. #3908 @pawel-big-lebowski
  Add retry logic for handling empty row exceptions.
• Spark: fix Spark version for databricks test. #3911 @pawel-big-lebowski
  Fix Spark version configuration in Databricks test environment.
• Flink: Fix connector of type kafka-upsert not identifying kafka topics correctly. #3915 @fetta
  Fix kafka-upsert connector to properly identify kafka topics.
• Airflow: Fail fast and reduce timeout for airflow tests. #3905 @kacpermuda
  Improve test performance by implementing fail-fast behavior and reduced timeouts.
• dbt: more telemetry, fix quadratic file reading. #3916 @mobuchowski
  Improve telemetry collection and fix performance issues with file reading.
• dbt: Fix dbt version. #3894 @mobuchowski
  Fix dbt version compatibility issues.
• Python: Fix filenames for windows users. #3889 @pawel-big-lebowski
  Fix filename handling to work correctly on Windows systems.
• Transport: Adjust log level when aliasing default_http transport. #3897 @dolfinus
  Adjust logging level for transport aliasing messages.
• Build: Improve comments and add some tests. #3901 @kacpermuda
  Improve code documentation and add additional test coverage.

Added

• Spark: Include spark_applicationDetails facet to all events #3848 @dolfinus
  Add spark_applicationDetails facet to all OpenLineage events emitted by the Spark integration
• Spark: Support additional facets #3850 @ddebowczyk92
  Adds support for additional facets in Spark integration
• Spark: disable connector by Spark config parameter #3880 @pawel-big-lebowski
  Add spark.openlineage.disabled entry to disable OpenLineage integration through Spark config parameters
• Spark: Fine-grained timeout config #3779 @pawel-big-lebowski
  Add extra timeout options to emit incomplete OpenLineage events in case of timeout when building facets. See buildDatasetsTimePercentage and facetsBuildingTimePercentage in docs for more details
• Python: Asynchronous HTTP transport implementation #3812 @mobuchowski
  Adds high-performance asynchronous HTTP transport with event ordering guarantees, configurable concurrency, and comprehensive error handling. Features START-before-completion event ordering, bounded queues, and real-time statistics
• dbt: Add DbtRun facet to dbt run events #3764 @dolfinus
  Adds DbtRun facet for tracking dbt run information
• Python: Add continue_on_success and sorting transport in CompositeTransport #3829 @kacpermuda
  Adds configuration options for CompositeTransport to control behavior and ordering
• Hive: Add jobType facet #3789 @dolfinus
  Adds jobType facet to Hive integration
• Hive: Add dialect=hive to SqlJobFacet #3863 @dolfinus
  Adds dialect field to SqlJobFacet for Hive integration
• Spec: SqlJobFacet now contains dialect #3819 @mobuchowski
  Adds dialect field to SqlJobFacet specification
• Spec: Formalize job naming #3826 @ddebowczyk92
  Formalizes job naming conventions in the specification
• Spec: Formalize dataset naming #3775 @ddebowczyk92
  Formalizes dataset naming conventions in the specification

Changed

• Spark: Use Hive as default Iceberg catalog implementation #3858 @dolfinus
  Updates Spark integration to use Hive as the default catalog implementation for Iceberg tables
• Spark: Replace weak hash-map with a map with weak keys and entries #3856 @ddebowczyk92
  Improves memory management in Spark integration by replacing weak hash-map implementation
• Spark: Support latest databricks runtime #3811 @pawel-big-lebowski
  Updates Spark integration to support the latest Databricks runtime
• Python: Remove transport.wait_for_completion() #3881 @dolfinus
  Removes wait_for_completion() method from Python transport interface
• Python: Reuse session in sync HttpTransport #3843 @dolfinus
  Improves performance by reusing HTTP sessions in synchronous transport
• Python: Implement Transport.close() for Datazone and Kinesis #3857 @dolfinus
  Adds proper cleanup methods for Datazone and Kinesis transports
• Python: Implement TransformTransport.close #3855 @dolfinus
  Adds cleanup method for TransformTransport
• Python: Implement KafkaTransport.wait_for_completion() and .close() #3838 @dolfinus
  Adds proper cleanup and completion methods for Kafka transport
• Java: Make CompositeTransport.close() more reliable #3841 @dolfinus
  Improves reliability of CompositeTransport cleanup process
• Java: Cover OpenLineageClient.close() with tests #3839 @dolfinus
  Adds test coverage for OpenLineageClient cleanup methods
• Java: Name threads used in Java client #3817 @mobuchowski
  Adds meaningful names to threads used in Java client for better debugging
• Flink: Close OpenLineageClient in onJobExecuted #3854 @dolfinus
  Ensures proper cleanup of OpenLineageClient in Flink 1.x integration
• Flink: Fixed a bug incorrectly loading configuration in Event Emitter #3799 @pan-siekierski
  Fixes configuration loading issue in Flink Event Emitter
• dbt: Make invocation_id field optional #3796 @dolfinus
  Makes invocation_id field optional in dbt integration
• dbt: More resiliency for missing dbt nodes #3836 @mobuchowski
  Improves error handling for missing dbt nodes
• Hive: Add docker-compose example for local testing #3800 @dolfinus
  Adds docker-compose setup for local Hive integration testing
• Airflow: Send pending events after Airflow DAG is finished #3849 @dolfinus
  Ensures all pending events are sent after DAG completion in Airflow integration

Fixed

• dbt: Fix log path, more precise file reading #3793 @mobuchowski
  Improves log file handling in dbt integration
• dbt: Fix deprecated configs #3859 @kacpermuda
  Replaces deprecated dbt configurations with current alternatives
• Spark: Fix missing .db suffix in database/namespace location name for BigQueryMetastoreCatalog #3874 @ddebowczyk92
  Fixes database naming issue in BigQuery Metastore catalog implementation
• Spark: Fix missing output's dataset catalog facet when running CTAS queries on Iceberg tables #3835 @ddebowczyk92
  Fixes missing catalog facet in output datasets for CTAS queries on Iceberg tables
• Spark: Delta merge with column #3871 @pawel-big-lebowski
  Fixes Delta merge operation handling with column-level lineage
• Spark: Support delta 3.3.2 #3861 @pawel-big-lebowski
  Adds support for Delta Lake version 3.3.2
• Spark: Call version utils method when dataset can be identified #3832 @pawel-big-lebowski
  Fixes version handling when dataset can be properly identified
• Java: Raise error if no events were emitted by composite transport #3853 @kacpermuda
  Adds error handling when CompositeTransport fails to emit any events
• Java: Fix IntelliJ reload #3825 @mobuchowski
  Fixes IntelliJ project reload issues
• Java: Fix spotless in hive integration #3806 @mobuchowski
  Fixes code formatting issues in Hive integration
• Java: Fix missing field in SQL facet test #3830 @mobuchowski
  Fixes missing field in SQL facet test case
• Java: Fix composite transport logging of multiple transports #3887 @kacpermuda
  Fixes logging issue when using CompositeTransport with multiple transports
• Rust: Fix new rust formatting warnings #3814 @mobuchowski
  Fixes formatting warnings in Rust code

Added

• Hive: Integration added. #3555 @tnazarew with @ddebowczyk92, @jphalip
  Added OpenLineage Hive integration
• Spark: Support dynamic frames. #3691 @pawel-big-lebowski
  Support lineage extraction from UnionRdd and NewHadoopRDD, which makes dynamic frames docker based test passing.
• Hive: Add hive_query facet #3781 @dolfinus
  Adds hive_query facet for Hive integration
• Hive: Add job sql facet #3777 @dolfinus
  Adds job sql facet for Hive integration
• Hive: Add hive session facet #3786 @dolfinus
  Adds hive session facet for Hive integration
• Java: Add Location Symlink type #3717 @tnazarew
  Add new symlink type representing physical location of dataset
• Spark: Smart debug facet. #3715 @pawel-big-lebowski
  Automatically turn on debug facet in case of spark connector anomalies detected
• Spark: Add support for Big Query Metastore catalog type #3760 @ddebowczyk92
  Adds support for BigQuery Metastore catalog in Spark integration
• dbt: Add DbtRun facet #3738 @dolfinus
  Adds DbtRun facet for tracking dbt run information
• dbt: Initial support for Clickhouse #3739 @dolfinus
  Adds initial support for ClickHouse in dbt integration
• dbt: Add processing_engine facet #3725 @dolfinus
  Adds processing_engine facet for dbt integration
• Flink: Add facet with Flink jobId #3744 @dolfinus
  Adds facet containing Flink job ID information
• Flink: Add processing_engine facet #3726 @dolfinus
  Adds processing_engine facet for Flink integration
• JDBC: Column level lineage for jdbc queries load #3763 @pawel-big-lebowski
  Adds column-level lineage support for JDBC queries for Spark with single input table
• Spec: Add contentType to documentation facet #3748 @dolfinus
  Adds contentType field to documentation facet specification

Changed

• Airflow: Remove Airflow < 2.5.0 support #3669 @kacpermuda
  Drops support for Airflow versions below 2.5.0
• dbt: Use adapter rows_affected as outputStatistics #3731 @dolfinus
  Uses adapter's rows_affected for output statistics instead of custom calculation
• dbt: Move facets from processor module #3713 @dolfinus
  Refactors dbt facets organization by moving them from processor module
• Java: Speedup generateNewUUID #3754 @dolfinus
  Improves performance of UUID generation in Java client
• Java: Make UUIDUtils.generateStaticUUID random part more variative #3709 @dolfinus
  Increases randomness in static UUID generation
• Java: Add log if load from yaml fails #3766 @mvitale
  Adds logging when YAML configuration loading fails
• Spark: Update Spark 4 dependency to 4.0.0 (remove -preview1 suffix) #3751 @ddebowczyk92
  Updates Spark 4 dependency to final 4.0.0 release
• Spark: Disable module metadata file generation #3785 @ddebowczyk92
  Disables generation of module metadata files in Spark integration
• Python: Use attr.define instead of attr.s #3776 @kacpermuda
  Modernizes Python code to use newer attrs API
• Proxy: Remove native proxy #3680 @mobuchowski
  Removes the native proxy implementation

Fixed

• Spark: Fix missing table path in InsertIntoHadoopFsRelationCommand #3773 @dolfinus
  Fixes issue where table path was missing in InsertIntoHadoopFsRelationCommand
• BigQuery: Filter temp inner jobs for bigquery indirect mode #3722 @pawel-big-lebowski
  Filters out temporary inner jobs in BigQuery indirect mode
• dbt: dbt-ol should not error on job complete if there is no start event #3749 @mobuchowski
  Prevents errors when job completion occurs without corresponding start event
• Flink: Do not hide OpenLineage config parsing errors #3724 @dolfinus
  Improves error visibility for OpenLineage configuration parsing in Flink
• Java: Prevent original events from being mutated in TransformTransport #3728 @JDarDagran
  Ensures original events remain immutable during transport transformations
• Java: Fix visibility of GcpLineageTransportConfig.Mode #3762 @ngorchakova
  Corrects visibility modifier for GCP transport configuration mode

Added

• Python: add TransformTransport for Python client #3697 @kacpermuda
  Introduces the TransformTransport class for event transformations.
• Spec: add CatalogDatasetFacet #3659 @mobuchowski
  Adds specification of the CatalogDatasetFacet.
• Spark: implement CatalogDatasetFacet #3695 @mobuchowski
  Implements CatalogDatasetFacet for Spark integration.
• Transport: add Amazon DataZone transport #3685 @shinabel
  Adds support for publishing events to Amazon DataZone via dedicated transport.
• JobNamespaceReplaceTransformer: modify parent run facet #3706 @kacpermuda
  Allows modifications of the parent run facet through JobNamespaceReplaceTransformer.

Changed

• dbt: remove empty facets #3700 @dolfinus
  Prevents empty facets from being emitted by dbt integration.
• Spark: broaden exception handling for Iceberg table writes #3686 @luke-hoffman1
  Broadens exception handling to address missing column-level lineage issues.

Fixed

• dbt: generate new runId for each start event #3696 @dolfinus
  Ensures dbt start events have unique run identifiers when calling id generation methods multiple times.
• dbt: skip database if not set #3707 @dolfinus
  Ensures database field is optional and handled gracefully.
• Spark: don't swallow exceptions in getIcebergTable #3688 @dolfinus
  Improves exception visibility during Iceberg table retrieval.
• Flink 2: support JDBC naming #3681 @pawel-big-lebowski
  Correctly handles JDBC dataset naming conventions in Flink 2 integration.

Added

• Avro: support schema facet for Avro datasets #3650 @pawel-big-lebowski
  This PR adds support for schema facets in Avro datasets.
• Java: add UUIDUtils.generateStaticUUID utility #3672 @martinovm
  Provides utility method for generating static UUIDs.

Fixed

• dbt: forward dbt's return code #3682 @MassyB
  Forwards dbt's process return code correctly.
• dbt: handle dbt log file rotation #3683 @MassyB
  Implements log rotation handling to avoid log file overflow.
• Flink2: fix dataset namespace resolvers #3676 @pawel-big-lebowski
  Correctly resolves dataset namespaces in Flink2.
• Spark: ensure Spark Job Name suffix extraction #3667 @luke-hoffman1
  Adds condition and unit test to correctly extract Spark Job Name suffix.
• Spark: fix NoSuchElementException when calling get() on Option.None #3673 @ddebowczyk92
  Prevents exceptions when Option.None is encountered.
• Spark: avoid InputDataset duplicates by skipping visited nodes #3663 @ddebowczyk92
  Ensures nodes aren't revisited, avoiding duplicate InputDataset creation.

Added

• Flink: enhance JDBC extractors with additional types #3652 @HuangZhenQiu

Changed

• Python: bump supported Python version to 3.9 #3648 @mobuchowski
• Java: generate Kinesis partitionKey using Run parent facet #3637 @dolfinus

Fixed

• Core: check for RunEvent v2 in filters and env_var_facet #3664 @mudakacper
• dbt: more robust handling of dbt CommandCompleted event #3661 @mobuchowski
• Spark: define equals for TransformedInput to avoid duplicates #3644 @tnazarew
• Flink: prevent initializing Flink2 listener for Flink1 #3645 @pawel-big-lebowski
• Spark: remove RepairTableCommandVisitor from Spark3VisitorFactoryImpl #3641 @ddebowczyk92

Added

• Java: added support for LDAP connection strings #3612 @luke-hoffman1
• dbt: added support for dbt-glue adapter #3625 @mehdimld
• Python: introduced CLI for facet generator #3605 @kuba0221
• spec: optional root parent run ID added to ParentRunFacet #3572 @mobuchowski

Changed

• Java: use parent.root facet as primary source for Kafka messageKey #3622 @dolfinus
• Python: use parent.root facet as primary source for Kafka messageKey #3624 @dolfinus

Fixed

• Flink2: fix job namespace retrieval from config #3621 @dolfinus
• Spark: extract column lineage from query string #3582 @ddebowczyk92
• Spark: fix duplicate input dataset creation when RelationV2 is a child of SubqueryAlias #3607 @ddebowczyk92
• Java: fix Oracle thin JDBC URL normalization #3586 @dolfinus
• Spark: fix internal metrics sent through DebugFacet. #3594 @pawel-big-lebowski Register simple micrometer registry when no other registry configured.

Fixed

• dbt: read structured log file incrementally #3580 @MassyB Buffered writing could cause dbt to write broken log lines. This PR handles that case.
• Python: fix user tags in event v2 events #3576 @pawel-big-lebowski This PR fixes case where event v2 job and run events did not get user tags.
• Spark: fix NullPointerException #3583 @ddebowczyk92 This PR fixes potential NullPointerException in SaveIntoDataSourceCommandVisitor.

Added

• Flink 2: added support for CheckpointFacet #3531 @pawel-big-lebowski Similar to Flink 1 integration, Flink 2 integration will emit CheckpointFacet.
• Flink 2: include SQL comments in OL events #3528 @pawel-big-lebowski Table and field comments are available within generated OL events.

Changed

• Java: remove deprecated configs: 'disabledFacets' and 'timeout'. #3522 @pawel-big-lebowski Configs have been replaced with: 'facets.facet-name.disabled=true' and 'timeoutInMillis'.

Fixed

• Spark: enable Iceberg metrics reporting for SparkSessionCatalog. #3538 @sakjung Fixes support for metrics in Iceberg SparkSessionCatalog.
• Spark: inject metrics reporter to Iceberg's RESTCatalog. #3515 @pawel-big-lebowski Fixes support for metrics in Iceberg RESTCatalog.
• dbt: fix race condition on dbt log file #3535 @MassyB Fixes race condition that was happening when using structured logs output.
• dbt: fixes handling of skipped dbt nodes #3545 @MassyB Skipped nodes no longed cause exceptions.
• Spark: fix record count stats mixed with bytes stats #3550 @pawel-big-lebowski InputStatistics facet for Iceberg datasets no longer produces incorrect stats.
• Spark: catch input duplicates for SubqueryAlias. #3548 @pawel-big-lebowski Subquery alias no longer is duplicating the inputs.
• Spark: catch Java 17 add opens exception. #3552 @mobuchowski Fixes catching InaccessibleMethodException in Java 17 within SparkExtensionVisitor.

New Contributors

• @sakjung made their first contribution in https://github.com/OpenLineage/OpenLineage/pull/3538
• @noklam made their first contribution in https://github.com/OpenLineage/OpenLineage/pull/3540

Added

• Python: allow adding user-supplied tags facets from config #3471 @leogodin217 User-supplied tags will allow the client to inject new tags or override tags provided by the integrations for jobs and runs.
• Java: allow adding user-supplied tags facets from config #3493 @mobuchowski Enabled parsing tags from config in Java client and Spark conf.

Changed

• Java: change async breaker timeout setting is not a real timeout. #3487 @pawel-big-lebowski Properly name case where TaskQueueCircuitBreaker allows a configurable blocking time after submitting a callable.
• Flink: enabled circuit breaker for Flink 2 integration. #3503 @pawel-big-lebowski Native Flink integration is now isolated within circuit breaker call.

Fixed

• Spark: use all of the underlying classloaders to find META-INF/services resources. #3483 @ddebowczyk92 ServiceLoader should not fail to load OpenLineageExtensionProvider implementations in certain configurations.
• Flink: handle default null job manager address. #3486 @MarquisC Null Flink Job Manager address will default to localhost
• dbt: handle tests on sources for structured logs option. #3488 @MassyB Handle case for tests on sources which don't have the attached_node defined in the manifest.

Added

• Java: enable specifying custom SSL context #3444 @pawel-big-lebowski Enable providing configuration for SSL context within HTTP transport.
• Spark: make Spark nodes filtering configurable. #3442 @pawel-big-lebowski Spark integration filters OpenLineage events for specific plan node classes. This can be now extended with extra config entries: allowedSparkNodes and deniedSparkNodes. See Spark Configuration documentation for more details.
• Java: add task queue based async circuit breaker. #3437 @aritrabandyo This circuit breaker that executes task on a queue backed threadpool, gives up tasks if the queue is full, and keeps track of rejected tasks.
• dbt: added initial support for Trino adapter #3429 @whitleykeith This allows Trino integration to emit proper events containing Trino datasets.
• Spark: increased coverage for Spark DML commands #3430 @ssanthanam185 Adds coverage for AlterTableRecoverPartitionsCommandVisitor, RefreshTableCommandVisitor, RepairTableCommandVisitor.

Changed

• Spark: the OpenLineageSparkListener was refactored to have a public, single-argument constructor taking an instance of SparkConf. #3425 @d-m-h This presents no functional change to the listener, however it will allow for improved initialisation of the listener in the future.
• Spark: Unsupported catalog exception should be less verbose. #3435 @pawel-big-lebowski In case of unsupported classes, warn logs without a stacktrace should be produced.
• Spark: Directly expose the LogicalPlan and SparkPlan objects inside OpenLineageContext. #3443 @d-m-h This is an initial refactor to a larger code base change that will see the removal of direct access of the QueryExecution object. It has no functional change on the way the integration behaves.

Fixed

• Spark: improve column lineage by including inputs within COMPLETE events. #3434 @pawel-big-lebowski *Send input datasets in COMPLETE events while making sure version facet is attached on START only.
• dbt: ParentRunFacet is now correctly attached when using structured logs option. #3432 @MassyB Fixes incorrect structure of ParentRunFacet.

Added

• Flink: Experimental version for flink native lineage listener. #3099 @pawel-big-lebowski New flink listener to extract lineage through native Flink interfaces. Supports Flink SQL. Requires Flink 2.0.
• dbt: add support for consuming dbt structured logs for test and build commands . #3362 @MassyB New option for dbt integration now can handle test and build commands too.
• Spark: allow attaching custom facets to RDDExecContext events. #3379 @ssanthanam185 Events emitted from RDDExecutionContext now include custom facets that get loaded as part of InternalHandlerFactory.
• spec: add DatasetTypeDatasetFacet. #3390 @JDarDagran DatasetTypeDatasetFacet allows explicit declaration of type of the resulting dataset.

Changed

• Python: allow adding additionalProperties to Python facets #3391 @JDarDagran Adds with_additonal_properties method that allows to create modified instance of facet with additional properties.
• Spark: allow attaching custom facets to RDDExecContext events. #3379 @ssanthanam185 Events emitted from RDDExecutionContext now include custom facets that get loaded as part of InternalHandlerFactory.
• Spark: SerializedFromObject events aren't filtered for not-delta plans. #3403 @ssanthanam185 Those events shouldn't be filtered outside Databricks/Delta ecosystem.

Fixed

• Spark: fixed ClassLoader handling for OpenLineageExtensionProvider. #3368 @ddebowczyk92 Fixes ClassNotFoundException issue when using the openlineage-spark integration alongside a Spark connector that implements the spark-extension-interfaces due to class loader conflicts.
• SQL: add minimal support for Snowflake LATERAL. #3368 @cisenbe SQL parser won't error on Snowflake's LATERAL keyword.
• dbt: handle errors in parse_assertions #3311 @dsaxton-1password dbt integration won't fail when looking at tests on seeds.
• Spark: fix infinite loop in RDD flatten & perf optimization. #3379 @ssanthanam185 Spark integration now correctly handles complex jobs that have cycles and nested RDD trees.
• Python: FileTransport now correctly attaches json file extension. #3404 @kacpermuda When append=False, the json file extension wasn't properly added before.

Added

• dbt: Consume dbt structured logs and report progress in real time. #3314 @MassyB If --consume-structured-logs flag is set, dbt integration will consume dbt structured logs and report execution progress in real time.
• Java: Add transform transport to allow event modification. #3301 @pawel-big-lebowski New transport type allows to modify the event based on the specified transformer class.
• Java: Parallel event emitting for composite transport. #3305[#3305] @pawel-big-lebowski Emit events in parallel for composite transport. Running in parallel is a default behaviour continueOnFailure set to true. Default value of continueOnFailure got changed from false to true.
• Spark: Collect ScanReport and CommitReport in OpenLineage events when dealing with Iceberg tables. #3256 @pawel-big-lebowski Collects additional Iceberg metrics for datasets read or written through the library. Visit Dataset Metrics docs for more details.
• dbt: add support for duckdb adapter #3280 @mobuchowski Adds support for duckdb adapter for dbt integration.

Changed

• Spark: Add DatasetFactory to support Dataset creation. #3207 @pawel-big-lebowski Adds DatasetFactory to support Dataset creation. This class is used to create Dataset instances for DatasetFactory.

Fixed

• Spark: fix inconsistent dataset naming #3285 @pawel-big-lebowski GCS path now has correctly stripped leading slash

Added

• Dbt: Add support for Column-Level Lineage in dbt integration. #3264 @mayurmadnani Dbt integration now uses SQL parser to add information about collected column-level lineage.
• Spark: Add input and output statistics about datasets read and written. #3240#3263 @pawel-big-lebowski Fix issues related to existing output statistics collection mechanism and fetch input statistics. Output statistics contain now amount of files written, bytes size as well as records written. Input statistics contain bytes size and number of files read, while record count is collected only for DataSourceV2 sources.
• Introduced InputStatisticsInputDatasetFacet #3238 @pawel-big-lebowski
  Extend spec with a new facet InputStatisticsInputDatasetFacet modelled after a similar OutputStatisticsOutputDatasetFacet to contain statistics about input dataset read by a job.

Changed

• Spark: Exclude META-INF/*TransportBuilder from Spark Extension Interfaces #3244 @tnazarew Excludes META-INF/*TransportBuilder to avoid version conflicts
• Spark: enables building input/output facets through DatasetFactory #3207 @pawel-big-lebowski
  Adds extra capabilities into DatasetFactory class, marks some public developers' API methods as deprecated.

Fixed

• dbt: fix compatibility with dbt v1.8 #3228 @NJA010 dbt integration now takes into account modified test_metadata field
• Spark: enabled Delta 3.x version compatibility #3253 @Jorricks
  Take into account modified initialSnapshot name

Added

• Spark: Add Dataproc run facet to include jobType property #3167 @codelixir
  Updates the GCP Dataproc run facet to include jobType property
• Add EnvironmentVariablesRunFacet to core spec #3186 @JDarDagran
  Use EnvironmentVariablesRunFacet in Python client
• Add assertions for format in test events #3221 @JDarDagran
• Spark: Add integration tests for EMR #3142 @arturowczarek
  Spark integration has integration tests for EMR

Changed

• Move Kinesis to separate module, migrate HTTP transport to httpclient5 #3205 @mobuchowski
  Moves Kinesis integration to a separate module and updates HTTP transport to use HttpClient 5.x
• Docs: Upgrade docusaurus to 3.6 #3219 @arturowczarek
• Spark: Limit the Seq size in RddPathUtils::extract() #3148 @codelixir
  Adds flag to limit the logs in RddPathUtils::extract() to avoid OutOfMemoryError for large jobs

Fixed

• Docs: Fix outdated Spark-related docs #3215 @mobuchowski
• Fix docusaurus-mdx-checker errors #3217 @arturowczarek
• [Integration/dbt] Parse dbt source tests #3208 @MassyB
  Fix: Consider dbt sources when looking for test results
• Avoid tests in configurable test #3141 @pawel-leszczynski

Added

• Java: added CompositeTransport #3039 @JDarDagran
  This allows user to specify multiple targets to which OpenLineage events will be emitted.
• Spark extension interfaces: support table extended sources #3062 @Imbruced
  Interfaces are now able to extract lineage from Table interface, not only RelationProvider.
• Java: added GCP Dataplex transport #3043 @ddebowczyk92
  Dataplex transport is now available as a separate Maven package for users that want to send OL events to GCP Dataplex
• Java: added Google Cloud Storage transport #3077 @ddebowczyk92
  GCS transport is now available as a separate Maven package for users that want to send OL events to Google Cloud Storage
• Java: added S3 transport #3129 @arturowczarek
  S3 transport is now available as a separate Maven package for users that want to send OL events to S3
• Java: add option to configure client via environment variables #3094 @JDarDagran
  Specified variables are now autotranslated to configuration values.
• Python: add option to configure client via environment variables #3114 @JDarDagran
  Specified variables are now autotranslated to configuration values.
• Python: add option to add custom headers in HTTP transport #3116 @JDarDagran
  Allows user to add custom headers, for example for auth purposes.
• Column level lineage: add full dataset dependencies #3097 #3098 @arturowczarek
  Now, if datasetLineageEnabled is enabled, and when column level lineage depends on the whole dataset, it does add dataset dependency instead of listing all the column fields in that dataset.
• Java: OpenLineageClient and Transports are now AutoCloseable #3122 @ddebowczyk92
  This prevents a number of issues that might be caused by not closing underlying transports

Fixed

• Python Facet generator does not validate optional arguments #3054 @JDarDagran
  This fixes issue where NominalTimeRunFacet Facet breaks when nominalEndTime is None
• SQL: report only actually used tables from CTEs, rather than all #2962 @Imbruced
  With this change, if SQL specified CTE, but does not use it in final query, the lineage won't be falsely reported
• Fluentd: Enhancing plugin's capabilities #3068 @jonathanlbt1
  This change enhances performance and docs of fluentd proxy plugin.
• SQL: fix parser to point to origin table instead of CTEs #3107 @Imbruced
  For some complex CTEs, parser emitted CTE as a target table instead of original table. This is now fixed.
• Spark: column lineage correctly produces for merge into command #3095 @Imbruced
  Now OL produces CLL correctly for the potential view in the middle.

Added

• SQL: add support for USE statement with different syntaxes #2944 @kacpermuda
  Adjusts our Context so that it can use the new support for this statement in the parser and pass it to a number of queries.
• Spark: add script to build Spark dependencies #3044 @arturowczarek
  Adds a script to rebuild dependencies automatically following releases.
• Website: versionable docs #3007 #3023 @pawel-big-lebowski
  Adds a GitHub action that creates a new Docusaurus version on a tag push, verifiable using the openlineage-site repo. Implements a monorepo approach in a new website directory.

Fixed

• SQL: add support for SingleQuotedString in Identifier() #3035 @kacpermuda
  Single quoted strings were being treated differently than strings with no quotes, double quotes, or backticks.
• SQL: support IDENTIFIER function instead of treating it like table name #2999 @kacpermuda
  Adds support for this identifier in SELECT, MERGE, UPDATE, and DELETE statements. For now, only static identifiers are supported. When a variable is used, this table is removed from lineage to avoid emitting incorrect lineage.
• Spark: fix issue with only one table in inputs from SQL query while reading from JDBC #2918 @Imbruced
  Events created did not contain the correct input table when the query contained multiple tables.
• Spark: fix AWS Glue jobs naming for RDD events #3020 @arturowczarek
  The naming for RDD jobs now uses the same code as SQL and Application events.

Added

• Spec: add GCP Dataproc facet #2987 @tnazarew Registers the Google Cloud Platform Dataproc run facet.

Fixed

• Airflow: update SQL integration code to work with latest sqlparser-rs main #2983 @kacpermuda
  Adjusts the SQL integration after our sqlparser-rs fork has been updated to the latest main.
• Spark: fix AWS Glue jobs naming for SQL events #3001 @arturowczarek
  SQL events now properly use the names of the jobs retrieved from AWS Glue.
• Spark: fix issue with column lineage when using delta merge into command #2986 @Imbruced
  A view instance of a node is now included when gathering data sources for input columns.
• Spark: minor Spark filters refactor #2990 @arturowczarek
  Fixes a number of minor issues.
• Spark: Iceberg tables in AWS Glue have slashes instead of dots in symlinks #2984 @arturowczarek
  They should use slashes and the prefix table/.
• Spark: lineage for Iceberg datasets that are present outside of Spark's catalog is now present #2937 @d-m-h Previously, reading Iceberg datasets outside the configured Spark catalog prevented the datasets from being present in the inputs property of the RunEvent.

Added

• Python: add CompositeTransport #2925 @JDarDagran
  Adds a CompositeTransport that can accept other transport configs to instantiate transports and use them to emit events.
• Spark: compile & test Spark integration on Java 17 #2828 @pawel-big-lebowski
  The Spark integration is always compiled with Java 17, while tests are running on both Java 8 and Java 17 according to the configuration.
• Spark: support preview release of Spark 4.0 #2854 @pawel-big-lebowski
  Includes the Spark 4.0 preview release in the integration tests.
• Spark: add handling for Window #2901 @tnazarew
  Adds handling for Window-type nodes of a logical plan.
• Spark: extract and send events with raw SQL from Spark #2913 @Imbruced
  Adds a parser that traverses QueryExecution to get the SQL query used from the SQL field with a BFS algorithm.
• Spark: support Mongostream source #2887 @Imbruced
  Adds a Mongo streaming visitor and tests.
• Spark: new mechanism for disabling facets #2912 @arturowczarek
  The mechanism makes FacetConfig accept the disabled flag for any facet instead of passing them as a list.
• Spark: support Kinesis source #2906 @Imbruced
  Adds a Kinesis class handler in the streaming source builder.
• Spark: extract DatasetIdentifier from extension LineageNode #2900 @ddebowczyk92
  Adds support for cases in which LogicalRelation has a grandChild node that implements the LineageRelation interface.
• Spark: extract Dataset from underlying BaseRelation #2893 @ddebowczyk92
  DatasetIdentifier is now extracted from the underlying node of LogicalRelation.
• Spark: add descriptions and Marquez UI to Docker Compose file #2889 @jonathanlbt1
  Adds the marquez-web service to docker-compose.yml.

Fixed

• Proxy: bug fixed on error messages descriptions #2880 @jonathanlbt1
  Improves error logging.
• Proxy: update Docker image for Fluentd 1.17 #2877 @jonathanlbt1
  Upgrades the Fluentd version.
• Spark: fix issue with Kafka source when saving with for each batch method #2868 @Imbruced
  Fixes an issue when Spark is in streaming mode and input for Kafka was not present in the event.
• Spark: properly set ARN in namespace for Iceberg Glue symlinks #2943 @arturowczarek
  Makes IcebergHandler support Glue catalog tables and create the symlink using the code from PathUtils.
• Spark: accept any provider for AWS Glue storage format #2917 @arturowczarek
  Makes the AWS Glue ARN generating method accept every format (including Parquet), not only Hive SerDe.
• Spark: return valid JSON for failed logical plan serialization #2892 @arturowczarek
  The LogicalPlanSerializer now returns <failed-to-serialize-logical-plan> for failed serialization instead of an empty string.
• Spark: extract legacy column lineage visitors loader #2883 @arturowczarek
  Refactors CustomCollectorsUtils for improved readability.
• Spark: add Kafka input source when writing in foreach batch mode #2868 @Imbruced
  Fixes a bug keeping Kafka input sources from being produced.
• Spark: extract DatasetIdentifier from SaveIntoDataSourceCommandVisitor options #2934 @ddebowczyk92
  Extracts DatasetIdentifier from command's options instead of relying on p.createRelation(sqlContext, command.options()), which is a heavy operation for JdbcRelationProvider.

Added

• Airflow: add log_url to AirflowRunFacet #2852 @dolfinus
  Adds taskinstance's log_url field to AirflowRunFacet.
• Spark: add handling for Generate #2856 @tnazarew
  Adds handling for Generate-type nodes of a logical plan (e.g., explode operations).
• Java: add DerbyJdbcExtractor #2869 @dolfinus
  Adds JdbcExtractor implementation for Derby database. As this is a file-based DBMS, its Dataset namespace is file and name is an absolute path to a database file.
• Spark: verify bytecode version of the built jar. #2859 @pawel-big-lebowski
  Extends the JarVerifier plugin to ensure all compiled classes have a bytecode version of Java 8 or lower.
• Spark: add Kafka streaming source support #2851 @d-m-h
  Adds support for Kafka streaming sources to Kafka streaming sinks. Inputs and outputs are now included in lineage events.

Fixed

• Airflow: replace datetime.now with airflow.utils.timezone.utcnow #2865 @kacpermuda
  Fixes missing timezone information in task FAIL events.
• Spark: remove shaded dependency in ColumnLevelLineageBuilder #2850 @tnazarew
  Removes the shaded Streams dependency in ColumnLevelLineageBuilder causing a ClassNotFoundException.
• Spark: make Delta dataset symlink consistent with non-Delta tables #2863 @dolfinus
  Makes dataset symlinks for Delta and non-Delta tables consistent.
• Spark: use Table's properties during column-level lineage construction #2855 @ddebowczyk92
  Fixes PlanUtils3 so Dataset identifier information based on a Table's properties is also retrieved during the construction of column-level lineage.
• Spark: extract job name creation to providers #2861 @arturowczarek
  The integration now detects if the spark.app.name was autogenerated by Glue and uses the Glue job name in such cases. Also, each job name provisioning strategy is now extracted to a separate provider.

Added

• Spark: configurable integration test #2755 @pawel-big-lebowski
  Provides command line tool capable of running Spark integration tests that can be created without Java.
• Spark: OpenLineage Spark extension interfaces without runtime dependency hell #2809 #2837 @ddebowczyk92
  New Spark extension interfaces without runtime dependency hell. Includes a test to verify the integration is working properly.
• Spark: support latest versions 3.4.3 and 3.5.1. #2743 @pawel-big-lebowski
  Upgrades CI workflows to run tests against latest Spark versions: 3.4.2 -> 3.4.3 and 3.5.0 -> 3.5.1.
• Spark: add extraction of the masking property in column-level lineage #2789 @tnazarew
  Adds extraction of the masking property during collection of dependencies for ColumnLineageDatasetFacet creation.
• Spark: collect table name from InsertIntoHadoopFsRelationCommand #2794 @dolfinus
  Collects a table name for INSERT INTO command for tables created with USING $fileFormat syntax, like USING orc.
• Spark, Flink: add PostgresJdbcExtractor #2806 @dolfinus Adds the default 5432 port to Postgres namespaces.
• Spark, Flink: add TeradataJdbcExtractor #2826 @dolfinus Converts JDBC URLs like jdbc:teradata/host/DBS_PORT=1024,DATABASE=somedb to datasets with namespace teradata://host:1024 and name somedb.table.
• Spark, Flink: add MySqlJdbcExtractor #2825 @dolfinus Handles different formats of MySQL JDBC URL, and produces datasets with consistent namespaces, like mysql://host:port.
• Spark, Flink: add OracleJdbcExtractor #2824 @dolfinus Handles simple Oracle JDBC URLs, like oracle:thin:@//host:port/serviceName and oracle:thin@host:port:sid, and converts each to a dataset with namespace oracle://host:port and name sid.schema.table or serviceName.schema.table.
• Spark: configurable test with Docker image provided #2822 @pawel-big-lebowski
  Extends the configurable integration test feature to enable getting the Docker image name as a name.
• Spark: Support Iceberg 1.4 on Spark 3.5.1. #2838 @pawel-big-lebowski
  Include Iceberg support for Spark 3.5. Fix column level lineage facet for UNION queries.
• Spec: add example for change in #2756 #2801 @Sheeri Updates the customLineage facet test for the new syntax created in #2756.

Changed

• Spark: fallback to spark.sql.warehouse.dir as table namespace #2767 @dolfinus
  In cases when a metastore is not used, falls back to spark.sql.warehouse.dir or hive.metastore.warehouse.dir as table namespace, instead of duplicating the table's location.

Fixed

• Java: handle dashes in hostname for JdbcExtractors #2830 @dolfinus
  Proper handling of dashes in JDBC URL hosts.
• Spark: fix Glue symlinks formatting bug #2807 @Akash2351 Fixes Glue symlinks with config parsing for Glue catalogid.
• Spark, Flink: fix DBFS namespace format #2800 @dolfinus Fixes the DBFS namespace format.
• Spark: fix Glue naming format #2766 @dolfinus
  Changes the AWS Glue namespace to match Glue ARN documentation.
• Spark: fix Iceberg dataset location #2797 @dolfinus
  Fixes Iceberg dataset namespace: instead of file:/some/path/database.table uses file:/some/path/database/table. For dataset TABLE symlink, uses warehouse location instead of database location.
• Spark: fix NPE and incorrect comment #2827 @pawel-big-lebowski Fixes an error caused by a recent upgrade of Spark versions that did not break existing tests.
• Spark: convert scheme and authority to lowercase in JdbcLocation #2831 @dolfinus
  Converts valid JDBC URL scheme and authority to lowercase, leaving intact instance/database name, as different databases have different default case and case-sensitivity rules.

Added

• Java: dataset namespace resolver feature #2720 @pawel-big-lebowski
  Adds a dataset namespace resolving mechanism that resolves dataset namespaces based on the resolvers configured. The core mechanism is implemented in openlineage-java and can be used within the Flink and Spark integrations.
• Spark: add transformation extraction #2758 @tnazarew
  Adds a transformation type extraction mechanism.
• Spark: add GCP run and job facets #2643 @codelixir
  Adds GCPRunFacetBuilder and GCPJobFacetBuilder to report additional facets when running on Google Cloud Platform.
• Spark: improve namespace format for SQLServer #2773 @dolfinus
  Improves the namespace format for SQLServer.
• Spark: verify jar content after build #2698 @pawel-big-lebowski
  Adds a tool to verify shadowJar content and prevent reported issues. These are hard to prevent currently and require manual verification of manually unpacked jar content.
• Spec: add transformation type info #2756 @tnazarew
  Adds information about the transformation type in ColumnLineageDatasetFacet. transformationType and transformationDescription are marked as deprecated.
• Spec: implementing facet registry (following #2161) #2729 @harels
  Introduces the foundations of the new facet Registry into the repo.
• Spec: register GCP common job facet #2740 @ngorchakova
  Registers the GCP job facet that contains common attributes that will improve the way lineage is parsed and displayed by the GCP platform. Based on the proposal, GCP Lineage would like to define facets that are expected from integrations. The list of support facets is not final and will be extended further by next PR.

Removed

• Java: remove deprecated localServerId option from Kafka config #2738 @dolfinus
  Removes localServerId from Kafka config, deprecated since 1.13.0.
• Java: remove deprecated Transport.emit(String) #2737 @dolfinus
  Removes Transport.emit(String) support, deprecated since 1.13.0.
• Spark: remove spark-interfaces-scala module #2781 @ddebowczyk92
  Replaces the existing spark-interfaces-scala interfaces with new ones decoupled from the Scala binary version. Allows for improved integration in environments where one cannot guarantee the same version of openlineage-java.

Changed

• Spark: add log info when emitting lineage from Spark (following #2650) #2769 @algorithmy1
  Enhances logging.

Fixed

• Flink: use namespace.name as Avro complex field type #2763 @dolfinus
  namespace.name is now used as Avro "type" of complex fields (record, enum, fixed).
• Java: repair empty dataset name #2776 @kacpermuda
  The dataset name should not be empty.
• Spark: fix events emitted for drop table for Spark 3.4 and above #2745 @pawel-big-lebowski @savannavalgi
  Includes dataset being dropped within the event, as it used to be prior to Spark 3.4.
• Spark, Flink: fix S3 dataset names #2782 @dolfinus
  Drops the leading slash from the object storage dataset name. Converts s3a:// and s3n:// schemes to s3://.
• Spark: fix Hive metastore namespace #2761 @dolfinus
  Fixes the dataset namespace for cases when the Hive metastore URL is set using $SPARK_CONF_DIR/hive-site.xml.
• Spark: fix NPE in column-level lineage #2749 @pawel-big-lebowski
  The Spark agent now checks to determine if cur.getDependencies() is not null before adding dependencies.
• Spark: refactor OpenLineageRunEventBuilder #2754 @pawel-big-lebowski
  Adds a separate class containing all the input arguments to call OpenLineageRunEventBuilder::buildRun.
• Spark: fix historyUrl format #2741 @dolfinus
  Fixes the historyUrl format in spark_applicationDetails.
• SQL: allow self-recursive aliases #2753 @mobuchowski
  Expressions like select * from test_orders as test_orders are now parsed properly.

Added

• Spark: add jobType facet to Spark application events #2719 @dolfinus Adds jobType facet to runEvents emitted by SparkListenerApplicationStart.

Fixed

• dbt: fix swapped namespace and name in dbt integration #2735 @JDarDagran Fixes variable names.
• Python: override debug level #2727 @mobuchowski Removes debug-level logging of HTTP requests.

Added

• Flink: handle Iceberg tables with nested and complex field types #2706 @dolfinus
  Creates SchemaDatasetFacet with nested fields for Iceberg tables with list, map and struct columns.
• Flink: handle Avro schema with nested and complex field types #2711 @dolfinus
  Creates SchemaDatasetFacet with nested fields for Avro schemas with complex types (union, record, map, array, fixed).
• Spark: add facets to Spark application events #2677 @dolfinus
  Adds support for Spark application start and stop events in the ExecutionContext interface.
• Spark: add nested fields to SchemaDatasetFieldsFacet #2689 @dolfinus
  Adds nested Spark Dataframe fields support to SchemaDatasetFieldsFacet. Also include field comment as description.
• Spark: add SparkApplicationDetailsFacet #2688 @dolfinus
  Adds SparkApplicationDetailsFacet to runEvents emitted on Spark application start.

Removed

• Airflow: remove Airflow < 2.3.0 support #2710 @kacpermuda
  Removes Airflow < 2.3.0 support.
• Integration: use v2 Python facets #2693 @JDarDagran
  Migrates integrations from removed v1 facets to v2 Python facets.

Fixed

• Spark: improve job suffix assigning mechanism #2665 @pawel-big-lebowski
  For some catalog handlers, the mechanism was creating different dataset identifiers on START and COMPLETE depending on whether a dataset was created or not. This improves the mechanism to assign a deterministic job suffix based on the output dataset at the moment of a start event. Note: this may change job names in some scenarios.
• Airflow: fix empty dataset name for AthenaExtractor #2700 @kacpermuda
  The dataset name should not be empty when passing only a bucket as S3 output in Athena.
• Flink: fix SchemaDatasetFacet for Protobuf repeated primitive types #2685 @dolfinus
  Fixes issues with the Protobuf schema converter.
• Python: clean up Python client code, add logging. #2653 @kacpermuda
  Cleans up client code, refactors logging in all Python modules.
• SQL: catch TokenizerErrors, PanicException #2703 @mobuchowski
  The SQL parser now catches and handles these errors.
• Python: suppress warning on importing v1 module in init.py. #2713 @JDarDagran
  Suppresses the deprecation warning when v1 facets are used.
• Integration/Java/Python: use UUIDv7 instead of UUIDv4 #2686 #2687 @dolfinus
  Uses UUIDv7 instead of UUIDv4 for runEvents. The new UUID version produces monotonically increasing values, which leads to more performant queries on the OL consumer side. Note: UUID version is an implementation detail and can be changed in the future.

Added

• Common/dbt: add DREMIO to supported dbt profile types #2674 @surisimran
  Adds support for dbt-dremio, resolving #2668.
• Flink: support Protobuf format for sources and sinks #2482 @pawel-big-lebowski
  Adds schema extraction from Protobuf classes. Includes support for nested object types, array type, map type, oneOf and any.
• Java: add facet conversion test #2663 @julienledem
  Adds a simple test that shows how to deserialize a facet in the server model.
• Spark: job type facet to distinguish RDD jobs from Spark SQL jobs #2652 @pawel-big-lebowski
  Sets the jobType property of JobTypeJobFacet to either SQL_JOB or RDD_JOB.
• Spark: add Glue symlink if reading from Glue catalog table #2646 @mobuchowski
  The dataset symlink now points to the Glue catalog table name if the Glue catalog table is used.
• Spark: add spark_jobDetails facet #2662 @dolfinus Adds a SparkJobDetailsFacet, capturing information about Spark application jobs -- e.g. jobId, jobDescription, jobGroup, jobCallSite. This allows for tracking an OpenLineage RunEvent with a specific Spark job in SparkUI.

Removed

• Airflow: drop old ParentRunFacet key #2660 @dolfinus Changes the integration to use the parent key for ParentFacet, dropping the outdated parentRun.
• Spark: drop SparkVersionFacet #2659 @dolfinus Drops the SparkVersion facet, deprecated since 1.2.0 and planned for removal since 1.4.0.
• Python: allow relative paths in URI formats for Python facets #2679 @JDarDagran
  Removes a URI validator that checked if scheme and netloc were present, allowing relative paths in URI formats for Python facets.

Changed

• GreatExpectations: rename ParentRunFacet key #2661 @dolfinus
  The OpenLineage spec defined the ParentRunFacet with the property name parent but the Great Expectations integration created a lineage event with parentRun. This renames ParentRunFacet key from parentRun to parent. For backwards compatibility, keep the old name.

Fixed

• dbt: support a less ambiguous logic to generate job names #2658 @blacklight Includes profile and models in the dbt job name to make it more unique.
• Spark: update to use org.apache.commons.lang3 instead of org.apache.commons.lang #2676 @harels
  Updates Apache Commons Lang to the latest version. We were mixing two versions, and the old one was not present in many places.

Added

• Java: allow timeout for circuit breakers #2609 @pawel-big-lebowski
  Extends the circuit breaker mechanism to contain a global timeout that stops running OpenLineage integration code when a specified amount of time has elapsed.
• Java: handle DataSetEvent and JobEvent in Transport.emit #2611 @dolfinus
  Adds overloads Transport.emit(OpenLineage.DatasetEvent) and Transport.emit(OpenLineage.JobEvent), reusing the implementation of Transport.emit(OpenLineage.RunEvent). Please note: Transport.emit(String) is now deprecated and will be removed in 1.16.0.
• Java/Python: add GZIP compression to HttpTransport #2603 #2604 @dolfinus
  Adds a compression option to HttpTransport config in the Java and Python clients, with gzip implementation.
• Java/Python/Proxy: properly set Kafka message key #2571 #2597 #2598 @dolfinus
  Adds a new messageKey option to KafkaTransport config in the Python and Java clients, as well as the Proxy. This option replaces the localServerId option, which is now deprecated. Default value is generated using the run id (for RunEvent), job name (for JobEvent) or dataset name (for DatasetEvent). This value is used by the Kafka producer to distribute messages along topic partitions, instead of sending all the events to the same partition. This allows for full utilization of Kafka performance advantages.
• Flink: add support for Micrometer metrics #2633 @mobuchowski Adds a mechanism for forwarding metrics to any Micrometer-compatible implementation for Flink as has been implemented for Spark. Included: MeterRegistry, CompositeMeterRegistry, SimpleMeterRegistry, and MicrometerProvider.
• Python: generate Python facets from JSON schemas #2520 @JDarDagran
  Objects specified with JSON Schema needed to be manually developed and checked in Python, leading to many discrepancies, including wrong schema URLs. This adds a datamodel-code-generator for parsing JSON Schema and generating Pydantic or dataclasses classes, etc. In order to use attrs (a more modern version of dataclasses) and overcome some limitations of the tool, a number of steps have been added in order to customize code to meet OpenLineage requirements. Included: updated references to the latest base JSON Schema spec for all child facets. Please note: newly generated code creates a v2 interface that will be implemented in existing integrations in a future release. The v2 interface introduces some breaking changes: facets are put into separate modules per JSON Schema spec file, some names are changed, and several classes are now kw_only.
• Spark/Flink/Java: support YAML config files together with SparkConf/FlinkConf #2583 @pawel-big-lebowski
  Creates a SparkOpenlineageConfig and FlinkOpenlineageConfig for a more uniform configuration experience for the user. Renames OpenLineageYaml to OpenLineageConfig and modifies the code to use only OpenLineageConfig classes. Includes a doc update to mention that both ways can be used interchangeably and final documentation will merge all values provided.
• Spark: add custom token provider support #2613 @tnazarew Adds a TokenProviderTypeIdResolver to handle both FQCN and (for backward compatibility) api_key types in spark.openlineage.transport.auth.type.
• Spark/Flink: support YAML config files together with SparkConf & FlinkConf approaches #2583 @pawel-big-lebowski
  Adds support for config entries being provided by both YAML file and integration-specific configuration (SparkConf/FlinkConf). Allows each integration to have its own config entries.
• Spark/Flink: job ownership facet #2533 @pawel-big-lebowski Enables configuration entries specifying ownership of the job that will result in an OwnershipJobFacet being attached to job facets.

Changed

• Java: sync Kinesis partitionKey format with Kafka implementation #2620 @dolfinus Changes the format of Kinesis partitionKey from {jobNamespace}:{jobName} to run:{jobNamespace}/{jobName} to match the Kafka transport implementation.

Fixed

• Python: make load_config return an empty dict instead of None when file empty #2596 @kacpermuda utils.load_config() now returns an empty dict instead of None in the case of an empty file to prevent an OpenLineageClient crash.
• Java: render lombok-generated methods in javadoc #2614 @dolfinus Fixes rendering of javadoc for methods generated by lombok annotations by adding a delombok step.
• Spark/Snowflake: parse NPE when query option is used and table is empty #2599 @mobuchowski
  Fixes NPE when using query option when reading from Snowflake.

Added

• Airflow: add lineage_job_namespace and lineage_job_name macros #2582 @dolfinus
  Adds new Airflow macros lineage_job_namespace(), lineage_job_name(task) that return an Airflow namespace and Airflow job name, respectively.
• Spec: Allow nested struct fields in SchemaDatasetFacet #2548 @dolfinus
  Adds nested fields support to SchemaDatasetFacet.

Fixed

• Spark: fix PMD for test #2588 @pawel-big-lebowski
  Clears pmdTestScala212 of warnings that clutter the logs.
• Dbt: propagate the dbt return code also when no OpenLineage events are emitted #2591 @blacklight
  dbt-ol now propagates the exit code of the underlying dbt process even if no lineage events are emitted.
• Java: make sure string isn't empty to prevent going out of bounds #2585 @harels
  String lookup was not accounting for empty strings and causing a java.lang.StringIndexOutOfBoundsException.
• Spark: use HashSet in column-level lineage instead of iterating through LinkedList #2584 @mobuchowski
  Takes advantage of performance gains available from using HashSet for collection.
• Python: fix missing pkg_resources module on Python 3.12 #2572 @dolfinus
  Removes pkg_resources dependency and replaces it with the packaging lib.
• Airflow: fix format returned by airflow.macros.lineage_parent_id #2578 @blacklight
  Fixes the run format returned by the lineage_parent_id Airflow macro and simplifies the format of the lineage_parent_id and lineage_run_id macros.
• Dagster: limit Dagster version to 1.6.9 #2579 @JDarDagran Adds an upper limit on supported versions of Dagster as the integration is no longer actively maintained and recent releases have introduced breaking changes.

Added

• Common: add support for SCRIPT-type jobs in BigQuery #2564 @kacpermuda In the case of SCRIPT-type jobs in BigQuery, no lineage was being extracted because the SCRIPT job had no lineage information - it only spawned child jobs that had that information. With this change, the integration extracts lineage information from child jobs when dealing with SCRIPT-type jobs.
• Spark: support for built-in lineage extraction #2272 @pawel-big-lebowski This PR adds a spark-interfaces-scala package that allows lineage extraction to be implemented within Spark extensions (Iceberg, Delta, GCS, etc.). The Openlineage integration, when traversing the query plan, verifies if nodes implement defined interfaces. If so, interface methods are used to extract lineage. Refer to the README for more details.
• Spark/Java: add support for Micrometer metrics #2496 @mobuchowski
  Adds a mechanism for forwarding metrics to any Micrometer-compatible implementation. Included: MeterRegistryyFactory, MicrometerProvider, StatsDMetricsBuilder, metrics config in OpenLineage config, and a Java client implementation.
• Spark: add support for telemetry mechanism #2528 @mobuchowski
  Adds timers, counters and additional instrumentation in order to implement Micrometer metrics collection.
• Spark: support query option on table read #2556 @mobuchowski Adds support for the Spark-BigQuery connector's query input option, which executes a query directly on BigQuery, storing the result in an intermediate dataset, bypassing Spark's computation layer. Due to this, the lineage is retrieved using the SQL parser, similarly to JDBCRelation.
• Spark: change SparkPropertyFacetBuilder to support recording Spark runtime #2523 @Ruihua98
  Modifies SparkPropertyFacetBuilder to capture the RuntimeConfig of the Spark session because the existing SparkPropertyFacet can only capture the static config of the Spark context. This facet will be added in both RDD-related and SQL-related runs.
• Spec: add fileCount to dataset stat facets #2562 @dolfinus
  Adds a fileCount field to DataQualityMetricsInputDatasetFacet and OutputStatisticsOutputDatasetFacet specification.

Fixed

• dbt: dbt-ol should transparently exit with the same exit code as the child dbt process #2560 @blacklight
  Makes dbt-ol transparently exit with the same exit code as the child dbt process.
• Flink: disable module metadata generation #2531 @HuangZhenQiu
  Disables the module metadata generation for Flink to fix the problem of having gradle dependencies to submodules within openlineage-flink.jar.
• Flink: fixes to version 1.19 #2507 @pawel-big-lebowski Fixes the class not found issue when checking for Cassandra classes. Also fixes the Maven pom dependency on subprojects.
• Python: small improvements to .emit() method logging & annotations #2539 @dolfinus
  Updates OpenLineage.emit debug messages and annotations.
• SQL: show error message when OpenLineageSql cannot find native library #2547 @dolfinus
  When the OpenLineageSql class could not load a native library, if returned None for all operations. But because the error message was suppressed, the user could not determine the reason.
• SQL: update code to conform to upstream sqlparser-rs changes #2510 @mobuchowski
  Includes tests and cosmetic improvements.
• Spark: fix access to active Spark session #2535 @pawel-big-lebowski
  Changes behavior so IllegalStateException is always caught when accessing SparkSession.
• Spark: fix Databricks environment #2537 @pawel-big-lebowski
  Fixes the ClassNotFoundError occurring on Databricks runtime and extends the integration test to verify DatabricksEnvironmentFacet.
• Spark: fixed memory leak in JobMetricsHolder #2565 @d-m-h The JobMetricsHolder#cleanUp(int) method now correctly purges unneeded state from both maps.
• Spark: fixed memory leak in UnknownEntryFacetListener #2557 @pawel-big-lebowski Prevents storing the state when a facet is disabled, purging the state after populating run facets.
• Spark: fix parsing JDBCOptions(table=...) containing subquery #2546 @dolfinus
  Prevents openlineage-spark from producing datasets with names like database.(select * from table) for JDBC sources.
• Spark/Snowflake: support query option via SQL parser #2563 @mobuchowski
  When a Snowflake job is bypassing Spark's computation layer, now the SQL parser will be used to get the lineage.
• Spark: always catch IllegalStateException when accessing SparkSession #2535 @pawel-big-lebowski IllegalStateException was not being caught.

Added

• Dagster: add new provider for version 1.6.10 #2518 @JDarDagran Adds the new provider required by the latest version of Dagster.
• Flink: support lineage for a hybrid source #2491 @HuangZhenQiu Adds support for hybrid source lineage for users of Kafka and Iceberg sources in backfill usecases.
• Flink: bump Flink JDBC connector version #2472 @HuangZhenQiu Bumps the Flink JDBC connector version to 3.1.2-1.18 for Flink 1.18.
• Java: add a OpenLineageClientUtils#loadOpenLineageJson(InputStream) and change OpenLineageClientUtils#loadOpenLineageYaml(InputStream) methods #2490 @d-m-h This improves the explicitness of the methods. Previously, loadOpenLineageYaml(InputStream) wanted the InputStream to contain bytes that represented JSON.
• Java: add info from the HTTP response to the client exception #2486 @davidjgoss Adds the status code and body as properties on the thrown exception when a non-success response is encountered in the HTTP transport.
• Python: add support for MSK IAM authentication with a new transport #2478 @mattiabertorello
  Eases publication of events to MSK with IAM authentication.

Removed

• Airflow: remove redundant information from facets #2524 @kacpermuda Refines the operator's attribute inclusion logic in facets to include only those known to be important or compact, ensuring that custom operator attributes with substantial data do not inflate the event size.

Fixed

• Airflow: proceed without rendering templates if task_instance copy fails #2492 @kacpermuda Airflow will now proceed without rendering templates if task_instance copy fails in listener.on_task_instance_running.
• Spark: fix the HttpTransport timeout #2475 @pawel-big-lebowski The existing timeout config parameter is ambiguous: implementation treats the value as double in seconds, although the documentation claims it's milliseconds. A new config param timeoutInMillis has been added. the Existing timeout has been removed from docs and will be deprecated in 1.13.
• Spark: prevent NPE if the context is null #2515 @pawel-big-lebowski Adds a check for a null context before executing end(jobEnd).
• Flink: fix class not found issue for Cassandra #2507 @pawel-big-lebowski Fixes the class not found issue when checking for Cassandra classes. Also fixes the Maven POM dependency on subprojects.
• Flink: refine the JDBC table name #2512 @HuangZhenQiu Enables the JDBC table name with a schema prefix.
• Flink: fix JDBC dataset naming #2508 @pawel-big-lebowski For JDBC, the Flink integration is not adjusted to the Openlineage naming convention. There is code that extracts the dataset namespace/name from the JDBC connection url, but it's in the Spark integration. As a solution, this code has to be extracted into the Java client and reused by the Spark and Flink integrations.
• Flink: fix failure due to missing Cassandra classes #2507 @pawel-big-lebowski Flink is failing when no Cassandra classes are present on the class path. This is happening because of CassandraUtils class which has a static hasClasses method, but it imports Cassandra-related classes in the header. Also, the Flink subproject contains an unnecessary maven-publish plugin.
• Flink: fix release runtime dependencies #2504 @HuangZhenQiu The shadow jar of Flink is not minimized, so some internal jars are listed as runtime dependences. This removes them from the final pom.xml file in the Flink module.
• Spec: improve Cassandra lineage metadata #2479 @HuangZhenQiu Following the namespace definition, we should use cassandra://host:port.

Added

• Airflow: add support for JobTypeJobFacet properties #2412 @mattiabertorello
  Adds support for Job type properties within the Airflow Job facet.
• dbt: add support for JobTypeJobFacet properties #2411 @mattiabertorello Adds support for Job type properties within the DBT Job facet.
• Flink: support Flink Kafka dynamic source and sink (https://github.com/OpenLineage/OpenLineage/pull/2417) @HuangZhenQiu Adds support for Flink Kafka Table Connector use cases for topic and schema extraction.
• Flink: support multi-topic Kafka Sink #2372 @pawel-big-lebowski Adds support for multi-topic Kafka sinks. Limitations: recordSerializer needs to implement KafkaTopicsDescriptor. Please refer to the limitations sections in documentation.
• Flink: support lineage for JDBC connector #2436 @HuangZhenQiu Adds support for use cases that employ this connector.
• Flink: add common config gradle plugin (https://github.com/OpenLineage/OpenLineage/pull/2461) @HuangZhenQiu Adds a common config gradle plugin to simplify the gradle files of Flink submodules.
• Java: extend circuit breaker loaded with ServiceLoader #2435 @pawel-big-lebowski Loads the circuit breaker builder with ServiceLoader as an addition to a list of implemented builders available within the existing package.
• Spark: integration now emits intermediate, application level events wrapping entire job execution #2371 @mobuchowski Previously, the Spark event model described only single actions, potentially linked only to some parent run. Closes #1672.
• Spark: support built-in lineage within DataSourceV2Relation #2394 @pawel-big-lebowski Enables built-in lineage extraction within from DataSourceV2Relation lineage nodes.
• Spark: add support for JobTypeJobFacet properties #2410 @mattiabertorello Adds support for Job type properties within the Spark Job facet.
• Spark: stop sending spark.LogicalPlan facet by default #2433 @pawel-big-lebowski spark.LogicalPlan has been added to default value of spark.openlineage.facets.disabled.
• Spark/Flink/Java: circuit breaker #2407 @pawel-big-lebowski Introduces a circuit breaker mechanism to prevent effects of over-instrumentation. Implemented within Java client, it serves both the Flink and Spark integration. Read the Java client README for more details.
• Spark: add the capability to publish Scala 2.12 and 2.13 variants of openlineage-spark #2446 @d-m-h Adds the capability to publish Scala 2.12 and 2.13 variants of openlineage-spark

Changed

• Spark: enable the app module to be compiled with Scala 2.12 and Scala 2.13 variants of Apache Spark (https://github.com/OpenLineage/OpenLineage/pull/2432) @d-m-h
  The spark.binary.version and spark.version properties control which variant to build.
• Spark: enable Scala 2.13 support in the app module #2432 @d-m-h Enables the app module to be built using both Scala 2.12 and Scala 2.13 variants of various Apache Spark versions, and enables the CI/CD pipeline to build and test them.
• Spark: don't fail on exception of UnknownEntryFacet creation #2431 @mobuchowski Failure to generate UnknownEntryFacet was resulting in the event not being sent.
• Spark: move Snowflake code into the vendor projects folders #2405 @mattiabertorello Creates a vendor folder to isolate Snowflake-specific code from the main Spark integration, enhancing organization and flexibility.

Fixed

• Flink: resolve PMD rule violation warnings #2403 @HuangZhenQiu Resolves the PMD rule violation warnings in the Flink integration module.
• Flink: Added the 'isReleaseVersion' property back to the build, enabling the Flink integration to be release #2468 @d-m-h The 'isReleaseVersion' property was removed from the build, preventing the Flink integration from being released.
• Python: fix issue with file config creating additional file #2447 @kacpermuda FileConfig was creating an additional file when not in append mode. Closes #2439.
• Python: fix issue with append option in file config #2441 @kacpermuda FileConfig was ignoring the append key in YAML config. Closes #2440
• Spark: fix integration catalog symlink without warehouse #2379 @algorithmy1 In the case of symlinked Glue Catalog Tables, the parsing method was producing dataset names identical to the namespace.
• Flink: fix IcebergSourceWrapper for Iceberg connector 1.17 #2409 @ensctom In Flink 1.17, the Iceberg catalogloader was loading the catalog in the open function, causing the loadTable method to throw a NullPointerException error.
• Spark: migrate spark35, spark3, shared modules to produce Scala 2.12 and Scala 2.13 variants #2390 #2385#2384 @d-m-h Migrates the three modules to use the refactored Gradle plugins. Also splits some tests into Scala 2.12- and Scala 2.13-specific versions.
• Spark: conform the spark2 module to the new build process #2391 @d-m-h
  Due to a change in the Scala Collections API in Scala 2.13, NoSuchMethodErrors were being thrown when running the openlineage-spack connector in an Apache Spark runtime compiled using Scala 2.13.

• Flink: support Flink 1.18 #2366 @HuangZhenQiu Adds support for the latest Flink version with 1.17 used for Iceberg Flink runtime and Cassandra Connector as these do not yet support 1.18.
• Spark: add Gradle plugins to simplify the build process to support Scala 2.13 #2376 @d-m-h
  *Defines a set of Gradle plugins to configure the modules and reduce duplication.
• Spark: support multiple Scala versions LogicalPlan implementation #2361 @mattiabertorello In the LogicalPlanSerializerTest class, the implementation of the LogicalPlan interface is different between Scala 2.12 and Scala 2.13. In detail, the IndexedSeq changes package from the scala.collection to scala.collection.immutable. This implements both of the methods necessary in the two versions.
• Spark: Use ScalaConversionUtils to convert Scala and Java collections #2357 @mattiabertorello This initial step is to start supporting compilation for Scala 2.13 in the 3.2+ Spark versions. Scala 2.13 changed the default collection to immutable, the methods to create an empty collection, and the conversion between Java and Scala. This causes the code to not compile between 2.12 and 2.13. This replaces the usage of direct Scala collection methods (like creating an empty object) and conversions utils with ScalaConversionUtils methods that will support cross-compilation.
• Spark: support MERGE INTO queries on Databricks #2348 @pawel-big-lebowski
  Supports custom plan nodes used when running MERGE INTO queries on Databricks runtime.
• Spark: Support Glue catalog in iceberg #2283 @nataliezeller1
  Adds support for the Glue catalog based on the 'catalog-impl' property (in this case we will not have a 'type' property).

Changed

• Spark: Move Spark 3.1 code from the spark3 project #2365 @mattiabertorello
  Moves the Spark 3.1-related code to a specific project, spark31, so the spark3 project can be compiled with any Spark 3.x version.

Fixed

• Airflow: add database information to SnowflakeExtractor #2364 @kacpermuda
  Fixes missing database information in SnowflakeExtractor.
• Airflow: add dag_id to task_run_id to avoid duplicates #2358 @kacpermuda
  The lack of dag_id in task_run_id can cause duplicates in run_id across different dags.
• Airflow: Add tests for column lineage facet and sql parser #2373 @kacpermuda
  Improves naming (database.schema.table) in SQLExtractor's column lineage facet and adds some unit tests.
• Spark: fix removePathPattern behaviour #2350 @pawel-big-lebowski
  The removepath pattern feature is not applied all the time. The method is called when constructing DatasetIdentifier through PathUtils which is not the case all the time. This moves removePattern to another place in the codebase that is always run.
• Spark: fix a type incompatibility in RddExecutionContext between Scala 2.12 and 2.13 #2360 @mattiabertorello
  The function from the ResultStage.func() object change type in Spark between Scala 2.12 and 2.13 makes the compilation fail. This avoids getting the function with an explicit type; instead, it gets it every time it is needed from the ResultStage object. This PR is part of the effort to support Scala 2.13 in the Spark integration.
• Spark: Fix removePathPattern feature #2350 @pawel-big-lebowski
  Refactors code to make sure that all datasets sent are processed through removePathPattern if configured to do so.
• Spark: Clean up the individual build.gradle files in preparation for Scala 2.13 support #2377 @d-m-h
  Cleans up the build.gradle files, consolidating the custom plugin and removing unused and unnecessary configuration.
• Spark: refactor the Gradle plugins to make it easier to define Scala variants per module #2383 @d-m-h
  The third of several PRs to support producing Scala 2.12 and Scala 2.13 variants of the OpenLineage Spark integration. This PR refactors the custom Gradle plugins in order to make supporting multiple variants per module easier. This is necessary because the shared module fails its tests when consuming the Scala 2.13 variants of Apache Spark.

COMPATIBILITY NOTICE Starting in 1.7.0, the Airflow integration will no longer support Airflow versions >=2.8.0.
Please use the OpenLineage Airflow Provider instead.

Added

• Airflow: add parent run facet to COMPLETE and FAIL events in Airflow integration #2320 @kacpermuda Adds a parent run facet to all events in the Airflow integration.

Fixed

• Airflow: repair up.sh for MacOS #2316 #2318 @kacpermuda Some scripts were not working well on MacOS. This adjusts them.
• Airflow: repair run_id for FAIL event in Airflow 2.6+ #2305 @kacpermuda The Run_id in a FAIL event was different than in the START event for Airflow 2.6+.
• Flink: open Iceberg TableLoader before loading a table #2314 @pawel-big-lebowski Fixes a potential NullPointerException in 1.17 when dealing with Iceberg sinks.
• Flink: name Kafka datasets according to the naming convention #2321 @pawel-big-lebowski Adds a kafka:// prefix to Kafka topic datasets' namespaces.
• Flink: fix properties within JobTypeJobFacet #2325 @pawel-big-lebowski Fixes properties assignment in the Flink visitor.
• Spark: fix commons-logging relocate in target jar #2319 @pawel-big-lebowski Avoids relocating a dependency that was getting excluded from the jar.
• Spec: fix inconsistency with Redshift authority format #2315 @davidjgoss Amends the Authority format for consistency with other references in the same section.

Removed

• Airflow: remove Airflow 2.8+ support #2330 @kacpermuda To encourage use of the Provider, this removes the listener from the plugin if the Airflow version is >=2.8.0.

Added

• Dagster: support Dagster 1.5.x #2220 @tsungchih Gets event records for each target Dagster event type to support Dagster version 0.15.0+.
• Dbt: add a new command dbt-ol send-events to send metadata of the last run without running the job #2285 @sophiely Adds a new command to send events to OpenLineage according to the latest metadata generated without running any dbt command.
• Flink: add option for Flink job listener to read from Flink conf #2229 @ensctom Adds option for the Flink job listener to read jobnames and namespaces from Flink conf.
• Spark: get column-level lineage from JDBC dbtable option #2284 @mobuchowski Adds support for dbtable, enables lineage in the case of single input columns, and improves dataset naming.
• Spec: introduce JobTypeJobFacet to contain additional job related information#2241 @pawel-big-lebowski New JobTypeJobFacet contains the processing type such as BATCH|STREAMING, integration via SPARK|FLINK|... and job type in QUERY|COMMAND|DAG|....
• SQL: add quote information from sqlparser-rs #2259 @JDarDagran Adds quote information from sqlparser-rs.

Fixed

• Spark: update Jackson dependency to resolve CVE-2022-1471 #2185 @pawel-big-lebowski Updates Gradle for Spark and Flink to 8.1.1. Upgrade Jackson 2.15.3.
• Flink: avoid relying on Guava which can be missing during production runtime #2296 @pawel-big-lebowski Removes usage of Guava ImmutableList.
• Spark: exclude commons-logging transitive dependency from published jar #2297 @pawel-big-lebowski Ensures commons-logging is not shipped as this can lead to a version mismatch on the user's side.

Added

• Flink: add Flink lineage for Cassandra Connectors #2175 @HuangZhenQiu Adds Flink Cassandra source and sink visitors and Flink Cassandra Integration test.
• Spark: support rdd and toDF operations available in Spark Scala API #2188 @pawel-big-lebowski
  Includes the first Scala integration test, fixes ExternalRddVisitor and adds support for extracting inputs from MapPartitionsRDD and ParallelCollectionRDD plan nodes.
• Spark: support Databricks Runtime 13.3 #2185 @pawel-big-lebowski Modifies the Spark integration to support the latest Databricks Runtime version.

Changed

• Airflow: loosen attrs and requests versions #2107 @JDarDagran Lowers the version requirements for attrs and requests and removes an unnecessary dependency.
• dbt: render yaml configs lazily #2221 @JDarDagran Don't render each entry in yaml files at start.

Fixed

• Airflow/Athena: change dataset name to its location #2167 @sophiely Replaces the dataset and namespace with the data's physical location for more complete lineage across integrations.
• Python client: skip redaction in column lineage facet #2177 @JDarDagran Redacted fields in ColumnLineageDatasetFacetFieldsAdditionalInputFields are now skipped.
• Spark: unify dataset naming for RDD jobs and Spark SQL #2181 @pawel-big-lebowski Use the same mechanism for RDD jobs to extract dataset identifier as used for Spark SQL.
• Spark: ensure a single START and a single COMPLETE event are sent #2103 @pawel-big-lebowski For Spark SQL at least four events are sent triggered by different SparkListener methods. Each of them is required and used to collect facets unavailable elsewhere. However, there should be only one START and COMPLETE events emitted. Other events should be sent as RUNNING. Please keep in mind that Spark integration remains stateless to limit the memory footprint, and it is the backend responsibility to merge several Openlineage events into a meaningful snapshot of metadata changes.

Added

• Client: allow setting client's endpoint via environment variable #2151 @mars-lan Enables setting this endpoint via environment variable because creating the client manually in Airflow is not possible.
• Flink: expand Iceberg source types #2149 @HuangZhenQiu
  Adds support for FlinkIcebergSource and FlinkIcebergTableSource for Flink Iceberg lineage.
• Spark: add debug facet #2147 @pawel-big-lebowski
  An extra run facet containing some system details (e.g., OS, Java, Scala version), classpath (e.g., package versions, jars included in the Spark job), SparkConf (like openlineage entries except auth, specified extensions, etc.) and LogicalPlan details (execution tree nodes' names) are added to events emitted. SparkConf setting spark.openlineage.debugFacet=enabled needs to be set to include the facet. By default, the debug facet is disabled.
• Spark: enable Nessie REST catalog #2165 @julwin
  Adds support for Nessie catalog in Spark.

Added

• Airflow: add some basic stats to the Airflow integration #1845 @harels
  Uses the statsd component that already exists in the Airflow codebase and wraps the section that emits to event with a timer, as well as emitting a counter for exceptions in sending the event.
• Airflow: add columns as schema facet for airflow.lineage.Table (if defined) #2138 @erikalfthan
  Adds columns (if set) from airflow.lineage.Table inlets/outlets to the OpenLineage Dataset.
• DBT: add SQLSERVER to supported dbt profile types #2136 @erikalfthan
  Adds support for dbt-sqlserver, solving #2129.
• Spark: support for latest 3.5 #2118 @pawel-big-lebowski
  Integration tests are now run on Spark 3.5. Also upgrades 3.3 branch to 3.3.3. Please note that delta and iceberg are not supported for Spark 3.5 at this time.

Fixed

• Airflow: fix find-links path in tox #2139 @JDarDagran
  Fixes a broken link.
• Airflow: add more graceful logging when no OpenLineage provider installed #2141 @JDarDagran
  Recognizes a failed import of airflow.providers.openlineage and adds more graceful logging to fix a corner case.
• Spark: fix bug in PathUtils' prepareDatasetIdentifierFromDefaultTablePath(CatalogTable) to correctly preserve scheme from CatalogTable's location #2142 @d-m-h
  Previously, the prepareDatasetIdentifierFromDefaultTablePath method would override the scheme with the value of "file" when constructing a dataset identifier. It now uses the scheme of the CatalogTable's URI for this. Thank you @pawel-big-lebowski for the quick triage and suggested fix.

Added

• Spark: publish the ProcessingEngineRunFacet as part of the normal operation of the OpenLineageSparkEventListener #2089 @d-m-h
  Publishes the spec-defined ProcessEngineRunFacet alongside the custom SparkVersionFacet (for now). The SparkVersionFacet is deprecated and will be removed in a future release.
• Spark: capture and emit spark.databricks.clusterUsageTags.clusterAllTags variable from databricks environment #2099 @Anirudh181001
  Adds spark.databricks.clusterUsageTags.clusterAllTags to the list of environment variables captured from databricks.

Fixed

• Common: support parsing dbt_project.yml without target-path #2106 @tatiana
  As of dbt v1.5, usage of target-path in the dbt_project.yml file has been deprecated, now preferring a CLI flag or env var. It will be removed in a future version. This allows users to run DbtLocalArtifactProcessor in dbt projects that do not declare target-path.
• Proxy: fix Proxy chart #2091 @harels
  Includes the proper image to deploy in the helm chart.
• Python: fix serde filtering #2044 @xli-1026
  Fixes the bug causing values in list objects to be filtered accidentally.
• Python: use non-deprecated apiKey if loading it from env variables @2029 @mobuchowski
  Changes api_key to apiKey in create_token_provider.
• Spark: Improve RDDs on S3 integration. #2039 @pawel-big-lebowski
  Prepares integration test to access S3, fixes input dataset duplicates and includes other minor fixes.
• Flink: prevent sending running events after job completes #2075 @pawel-big-lebowski
  Flink checkpoint tracking thread was not getting stopped properly on job complete.
• Spark & Flink: Unify dataset naming from URI objects #2083 @pawel-big-lebowski
  Makes sure Spark and Flink generate same dataset identifiers for the same datasets by having a single implementation to generate dataset namespace and name.
• Spark: Databricks improvements #2076 @pawel-big-lebowski
  Filters unwanted events on databricks and adds an integration test to verify this. Adds integration tests to verify dataset naming on databricks runtime is correct when table location is specified. Adds integration test for wide transformation on delta tables.

Removed

• SQL: remove sqlparser dependency from iface-java and iface-py #2090 @JDarDagran
  Removes the dependency due to a breaking change in the latest release of the parser.