pyspark

This project publishes no release notes. Between v4.1.3 and v4.2.0 there were 250 commits, 250 of them substantive:

• [SPARK-48091][SQL] Preserve aliases inside lambda when ExtractGenerator restructures plan
• [SPARK-56870][SDP] Implement SCD1 Batch Processor; Extend Microbatch with CDC Metadata
• [SPARK-56998] Add SECURITY.md + AGENTS.md Security section for scan-agent discoverability
• [SPARK-56882][SDP] Implement SCD1 Batch Processor; Target Column Projection
• [SPARK-56249][SDP] Implement SCD1 Batch Processor; Merge Tombstones onto Microbatch
• [SPARK-56719][SS] Add DataStreamWriter.name() API for sink evolution
• [SPARK-56961][SQL] Pass all options while loading changelog
• [SPARK-55846][DOCS] Update Web UI documentation for UI modernization
• [SPARK-56953][SDP] Implement SCD1 Batch Processor; foreachBatch Callback
• [SPARK-56697][SQL][DML] Refactor Catalog Manager
• [SPARK-55978][SQL][FOLLOWUP] Don't block V2 join pushdown when pushed Sample has fraction=1 without replacement
• [SPARK-55601][SS][FOLLOWUP] Cache offsetLog.getLatest() to avoid redundant ListStatus during MicroBatchExecution init
• [SPARK-56921][SQL] Fix CTE ID normalization for nested CTEs
• [SPARK-56643][CONNECT][TEST] Add DSv2 temp view with stored plan tests using Spark Connect mode
• [SPARK-56956][SDP] Introduce AutoCDC Flow Dataclasses
• [SPARK-56651][CONNECT][SDP] Add Python APIs for Auto CDC SCD Type 1
• [SPARK-57041][SQL] Fix deadlock between waitForSubqueries and lazy val initialization in subquery
• [SPARK-57040][SQL] JDBC connector supports pushdown TABLESAMPLE SYSTEM
• [SPARK-57037][CORE][TESTS] Force GC before allocating large array in SorterSuite to fix flaky OOM
• [SPARK-56619][SQL][TEST] Add DSv2 repeated table access tests with internal/external changes in Classic/Connect mode

…and 230 more.

This project publishes no release notes. Between v4.1.2 and v4.1.3 there were 109 commits, 109 of them substantive:

• Revert "Removing test jars and class files"
• Preparing development version 4.1.3-SNAPSHOT
• [SPARK-56448][CONNECT] Fix NPE on Spark Connect client restart due to ammonite compile cache
• [SPARK-56592][K8S] Skip upgrading setuptools in K8s PySpark Dockerfile
• [SPARK-56998] Add SECURITY.md + AGENTS.md Security section for scan-agent discoverability
• [SPARK-55897][SQL][4.1] Handle UserDefinedType in ColumnarRow, ColumnarBatchRow, and ColumnarArray get()
• [SPARK-57098][UI] Worker UI JSON endpoint redaction
• [SPARK-57116][SQL][PYTHON][DOC] Fix versionadded/@since for kll_merge_agg_* (4.1.0 -> 4.1.2)
• [SPARK-57115][INFRA] Add Java 17 workflow file
• [SPARK-57239][PYTHON][TEST] Use sqlite uri for mlflow model
• [SPARK-57297][SQL][TESTS] Add a test that SQL execution description respects spark.sql.redaction.string.regex
• [SPARK-57254][INFRA] Put CI-unrelated files in a module so CI won't be triggered
• [SPARK-57267][4.1][INFRA] Prepare branch-4.1 build workflows for decoupling
• [SPARK-57262][SQL][WEBUI][4.1] Job description derived from a query should respect spark.sql.redaction.string.regex
• [MINOR][PYTHON][DOC] Fix broken See Also links in pyspark.sql.functions
• [SPARK-57348][PYTHON][TESTS] Replace sql_keywords doctest show() with columns check
• [SPARK-57325][CONNECT] Stop streaming queries registered while the Connect session is closing
• [SPARK-57383][4.1][SQL][PYTHON] Honor configured Arrow zstd compression level when writing Arrow batches
• [SPARK-57393] Build: PySpark and SparkR source distributions are missing LICENSE and NOTICE files
• [SPARK-57397][PYTHON] Raise an exception when parsed UDT is the wrong type

…and 89 more.

This project publishes no release notes. Between v4.1.1 and v4.1.2 there were 109 commits, 109 of them substantive:

• Revert "Removing test jars and class files"
• Preparing development version 4.1.2-SNAPSHOT
• [SPARK-54886] Add base session created in SparkConnectService
• [SPARK-49872][FOLLOWUP] Remove Jackson JSON string length limit in KVStoreScalaSerializer
• [SPARK-54852][SQL] NOT IN subquery returns incorrect results with a collated table
• [SPARK-53991][SQL][TEST][FOLLOW-UP] Improve KLL quantile errors by removing internal details
• [SPARK-54982][INFRA] Use ASF_NEXUS_TOKEN to release and drop staging repository when finalizing the RC
• [SPARK-54610][SDP][DOCS] Improve SDP Programming Guide
• [SPARK-55005][SQL] Fix CONTINUE HANDLER to continue loop execution after handling exceptions in loop body
• [SPARK-54979][BUILD] Upgrade ORC to 2.2.2
• [SPARK-54785][SQL] Add support for binary sketch aggregations in KLL
• [SPARK-54886] Lazily create Spark Connect reuse session
• [SPARK-55015][SS][SQL] Fix decodeRemainingKey numFields calculation in PrefixKeyScanStateEncoder
• [SPARK-43986][SQL][FOLLOWUP] Catch ArrayIndexOutOfBoundsException in Datasketches HLL functions
• [SPARK-55070][SQL][CONNECT] Allow hidden column in dataframe column resolution
• [SPARK-55033][SQL] Fix stringArgs of DSv2 writing commands
• [SPARK-55096][PYTHON] Update pandas minimum version in connect/setup.py
• [SPARK-55097][SQL] Fix re-adding cached artifacts drops blocks silently issue
• [SPARK-55071][CONNECT][PYTHON] Make spark.addArtifact work with Windows paths
• [SPARK-55098][PYTHON] Vectorized UDFs with output batch size control fail with memory leak

…and 89 more.

This project publishes no release notes. Between v4.1.0 and v4.1.1 there were 50 commits, 50 of them substantive:

• Revert "Removing test jars and class files"
• Preparing development version 4.1.1-SNAPSHOT
• [SPARK-54695][TEST][CORE] StandaloneDynamicAllocationSuite.syncExecutors should ensure executors have fully setup
• [SPARK-7008][SDP][ML] Make org.apache.spark.sql.pipelines internal package and make EstimatorUtils private
• [SPARK-54683][SQL] Unify geo and time types blocking
• [SPARK-54692][DOCS] Add python_worker_logs tvf doc to API reference
• [SPARK-53991][SQL][FOLLOWUP] Enforce KLL_SKETCH_AGG_GET_RANK/QUANTILE arguments are foldable
• [SPARK-54686][SQL] Relax DSv2 table checks in temp views to allow new top-level columns
• [SPARK-54696][CONNECT] Clean-up ArrowBuffers in Connect
• [SPARK-53991][SQL][FOLLOWUP] Remove wrong comments added in PR #53463
• [SPARK-54728][PYTHON][DOCS] Remove a wrong note in dataframe.isEmpty
• [SPARK-52819][SQL] Making KryoSerializationCodec serializable to fix java.io.NotSerializableException errors in various use cases
• [SPARK-54730][SQL][CONNECT] Delay failure of dataframe column resolution
• [SPARK-54746][DOC] Fix wrong link in SQL Scripting page
• [SPARK-54621][SQL] Merge Into Update Set * preserve nested fields if …coerceNestedTypes is enabled
• [SPARK-46741][SQL] Cache Table with CTE won't work
• [SPARK-54749][SQL] Fix incorrect numOutputRows metric in OneRowRelationExec
• [SPARK-54652][SQL] Complete conversion of IDENTIFIER()
• [SPARK-54696][CONNECT] Clean-up Arrow Buffers - follow-up
• [SPARK-54556][CORE] Rollback succeeding shuffle map stages when shuffle checksum mismatch detected

…and 30 more.

This project publishes no release notes. Between v4.0.4 and v4.1.0 there were 250 commits, 250 of them substantive:

• [SPARK-54176][GEO][PYTHON] Introduce Geography and Geometry data types to PySpark Connect
• [SPARK-54079][GEO][SQL][FOLLOWUP] Fix catalyst expressions ST package and imports
• [SPARK-54160][GEO][SQL] Implement the ST_SetSrid expression in SQL
• [SPARK-54178][SQL] Improve error for ResolveSQLOnFile
• [SPARK-54197][K8S] Improve ExecutorsPodsLifecycleManager not to request to delete if deletionTimestamp exists
• [SPARK-54159][INFRA] Fix pip install commands to use proper quotation marks
• [SPARK-54213][PYTHON][CONNECT] Remove Python 3.9 from Spark Connect
• Revert "[SPARK-53525][CONNECT][FOLLOWUP] Spark Connect ArrowBatch Result Chunking - Scala Client"
• [SPARK-54101][GEO][FOLLOWUP] Re-enable Scala/Python parity check for ST functions
• [SPARK-54200][SS] Call close() against underlying InputPartition when LowLatencyReaderWrap.close() is called
• [SPARK-54177][BUILD][FOLLOWUP] Restore Guava exclusion from connect-common shading artifacts
• [SPARK-54198][K8S] Delete Kubernetes executor pods only once per event processing interval
• [SPARK-54169][GEO][SQL] Introduce Geography and Geometry types in Arrow writer
• [SPARK-52463][SDP] Add support for cluster_by in Python Pipelines APIs
• [SPARK-53944][K8S][CORE][FOLLOWUP] Fix SparkEnv to use DRIVER_BIND_ADDRESS in case of useDriverPodIP=true
• [MINOR][DOCS][CONNECT] Replace pyspark with pyspark-client in the Connect overview
• [SPARK-54201][GEO][SQL] Allow casting from GeographyType(srid) to GeographyType(ANY)
• [SPARK-54190][BUILD] Guava dependency governance
• [SPARK-54226][SQL] Extend Arrow compression to Pandas UDF
• [SPARK-54202][GEO][SQL] Allow casting from GeometryType(srid) to GeometryType(ANY)

…and 230 more.

This project publishes no release notes. Between v4.0.3 and v4.0.4 there were 83 commits, 83 of them substantive:

• Revert "Removing test jars and class files"
• Preparing development version 4.0.4-SNAPSHOT
• [SPARK-57254][INFRA] Put CI-unrelated files in a module so CI won't be triggered
• [SPARK-57262][SQL][WEBUI][4.0] Job description derived from a query should respect spark.sql.redaction.string.regex
• [SPARK-57348][PYTHON][TESTS][4.0] Replace sql_keywords doctest show() with columns check
• [SPARK-57393] Build: PySpark and SparkR source distributions are missing LICENSE and NOTICE files
• [SPARK-57397][PYTHON] Raise an exception when parsed UDT is the wrong type
• [SPARK-57398][4.0][INFRA] Make branch-4.0 scheduled build workflows self-contained
• [SPARK-57429][SQL] Revise debug message in ThriftHttpServlet
• [SPARK-57426][CORE] Validate length before allocation in Encoders.Strings.decode
• [SPARK-57430][CORE] Validate length before allocation in Encoders.ByteArrays.decode
• [SPARK-57439][CORE] Validate length before allocation in Encoders.(Int|Long)Arrays decode
• [SPARK-57446][SQL] Apply escapeSql to comment in JDBC table/schema comment queries
• [SPARK-57447][SQL] Escape the index name in (H2|MySQL|Postgres)Dialect.indexExists
• [SPARK-57451][CORE] Remove the check rejecting spark.authenticate.secret with Master REST server
• [SPARK-57498][SQL] Strengthen cookie-signing secret in ThriftHttpServlet
• [SPARK-57474][CORE] Validate length before allocation in Encoders.StringArrays.decode
• [SPARK-57475][CORE] Validate length before allocation in Encoders.BitmapArrays.decode
• [SPARK-56724][INFRA][4.0] Make docker/* GitHub Actions up-to-date
• [SPARK-57509][CORE] Return FORBIDDEN instead of METHOD_NOT_ALLOWED for disallowed /workers/kill requests

…and 63 more.

This project publishes no release notes. Between v4.0.2 and v4.0.3 there were 70 commits, 70 of them substantive:

• Revert "Removing test jars and class files"
• Preparing development version 4.0.3-SNAPSHOT
• [SPARK-55387][CORE][UI] Fix DAG visualization not rendering due to malformed DOT label
• [SPARK-55423][INFRA] Set strategy.max-parrallel to 20 for all GitHub Action jobs
• [SPARK-55434][INFRA] Add username and password at svn with rm at finalize step
• [SPARK-55128][INFRA][4.0] Restore SQL tests by pin 'pandas==2.3.3'
• [SPARK-55411][SQL][4.0] SPJ may throw ArrayIndexOutOfBoundsException when join keys are less than cluster keys
• [SPARK-55495][CORE] Fix EventLogFileWriters.closeWriter to handle checkError
• [SPARK-55497][BUILD][TESTS] Use jupyterTestFramework instead of TestFrameworks.JUnit
• [SPARK-55599][BUILD] Replaces dots to hyphens in release announcement pages
• [SPARK-54599][PYTHON][FOLLOWUP][4.0] Remove an error message check to make CI happy
• [SPARK-55416][SS][PYTHON][4.0] Streaming Python Data Source memory leak when end-offset is not updated
• [SPARK-54599][PYTHON][FOLLOWUP][4.0] Remove another error message check to make CI happy
• [SPARK-55302][SQL][4.0] Fix custom metrics in case of KeyGroupedPartitioning
• [SPARK-55619][SQL][4.0] Fix custom metrics in case of coalesced partitions
• [SPARK-55637][SQL][TESTS] Generalize postgres-krb-setup.sh to find config files
• [SPARK-55688][BUILD] Upgrade aircompressor to 2.0.3
• [SPARK-55692][SQL] Fix SupportsRuntimeFiltering and SupportsRuntimeV2Filtering documentation
• [SPARK-55746][SQL][TESTS] Fix unable to load custom metric object SupportedV1WriteMetric
• [SPARK-55802][SQL][4.0] Fix integer overflow when computing Arrow batch bytes

…and 50 more.

This project publishes no release notes. Between v4.0.1 and v4.0.2 there were 84 commits, 84 of them substantive:

• Revert "Removing test jars and class files"
• Preparing development version 4.0.2-SNAPSHOT
• [MINOR][BUILD] Fix download of preview releases in the news
• [MINOR][BUILD] Remove todos for testing in the real releases
• [MINOR][BUILD] Remove preview postfix in documentation.md when releasing
• [SPARK-53472][DOCS] Fix jekyll-redirect-from template and generated html files
• [SPARK-53505][CONNECT] Try the eager analysis first for withColumns and withColumnsRenamed in Spark Connect
• [SPARK-53505][CONNECT][FOLLOWUP][4.0] Import SparkException in SparkConnectPlanner.scala to make build success
• [SPARK-50888][CONNECT][TESTS][4.0] Remove flaky assertion from SparkConnectServiceSuite
• [SPARK-53527][SQL] Improve fallback of analyzeExistenceDefaultValue
• [SPARK-53518][SQL][4.0] No truncation for catalogString of User Defined Type
• [SPARK-48163][CONNECT][TESTS][4.0] Fix flaky tests in SparkConnectServiceSuite which are caused by executorHolder undefined
• [SPARK-53434][SQL][4.0] ColumnarRow's get should also check isNullAt
• [SPARK-53557][INFRA] Reduce automated vote email deadline from 4 days to 73 hours
• [SPARK-53538][SQL][4.0] ExpandExec should initialize the unsafe projections
• [SPARK-53524][CONNECT][SQL][4.0] Fix temporal value conversion in LiteralValueProtoConverter
• [SPARK-53539][INFRA][4.0] Add libwebp-dev to recover spark-rm/Dockerfile building
• [SPARK-53577][DOCS] Fix Scaladoc source links for java sources
• [SPARK-53560][SS][SQL] Crash looping when retrying uncommitted batch in Kafka source and AvailableNow trigger
• [SPARK-52601][SQL][4.0] Support primitive types in TransformingEncoder

…and 64 more.

This project publishes no release notes. Between v4.0.0 and v4.0.1 there were 133 commits, 133 of them substantive:

• Revert "Removing test jars and class files"
• Preparing development version 4.0.1-SNAPSHOT
• [SPARK-52225][BUILD][4.0][3.5] Avoid using interactive TTY mode in release scripts for branch-3.5 and 4.0
• [SPARK-52178][BUILD][FOLLOW-UP] Respect ANSWER for checking tags
• [SPARK-52178][BUILD][FOLLOW-UP] Pass ANSWER environment variable to Docker container
• [SPARK-52231][BUILD] Propagate GITHUB_ACTIONS environment variable in release script
• [SPARK-52181][SQL][FOLLOWUP][4.0] Avoid OOM in variant tests
• [SPARK-52241][BUILD] Upload to repository in parallel in release script
• [SPARK-52237][DOCS] Fix the documentation of hypot function
• [SPARK-52241][BUILD][FOLLOW-UP] Make it fail if one of the curls fails
• [SPARK-52255][BUILD] Adds a new environment variable SPARK_RC_COUNT in release script
• [SPARK-52258][BUILD] Increase the number of retries and decrease the parallelism in the release script
• [SPARK-52258][BUILD][FOLLOW-UP] Add retries when connecting to the ASF repo
• [SPARK-50137][SQL] Avoid fallback to Hive-incompatible ways when table creation fails by thrift exception
• [SPARK-52153][SQL] Fix from_json and to_json with variant
• [SPARK-52259][ML][CONNECT] Fix Param class binary compatibility
• [SPARK-52225][BUILD][FOLLOW-UP] Change -it to -ti in Docker execution in release script
• [SPARK-52276][BUILD] Make ANSWER not to be overwritten in release scripts
• [SPARK-51430][PYTHON] Stop PySpark context logger from propagating logs to stdout
• [SPARK-52287][CORE] Improve SparkContext not to populate o.a.s.internal.io.cloud.*-related setting if not exist

…and 113 more.