clickhouse-connect

Bug Fix

• Fix insert of UUID strings including dashes. Closes #368

Bug Fix

• When using query_df with a FixedString column with a read format of 'string' (and the default query_df setting use_extended_dtypes=True), the resulting column in the dataframe will now be correctly set to the (extended) String dtype. Fixes https://github.com/ClickHouse/clickhouse-connect/issues/356

Improvement

• Python or Pandas float value to ClickHouse Decimal now correctly rounds Float values for more accurate conversions. Thanks to Frederik Eychenié for the investigation and PR!

Bug Fix

• Clean up pandas series concatenation issue

Bug Fixes

• query_df would raise a deprecation warning with recent Pandas version if there were empty blocks. This should be fixed. https://github.com/ClickHouse/clickhouse-connect/issues/349
• avoid a warning in timezone handling using the tzlocal library. Thanks to Tanner for the fix

Improvement

• The new client keyword argument show_clickhouse_errors controls whether the full ClickHouse error (including possibly sensitive information) is displayed when there is an error in ClickHouse processing. It defaults to True. If False, the simple string 'The ClickHouse server returned an error.' will be displayed. Closes https://github.com/ClickHouse/clickhouse-connect/issues/344.
• Updated to Cython 3.0.10

Improvement

Breaking Change

• The default behavior of applying the client timezone if the GMT offset of the client matched the GMT offset of the server for the current time has been changed. The new default is to always apply the server timezone unless the optional apply_server_timezone get_client parameter is explicitly set to False. The previous behavior could cause confusing results where datetime values would be rendered in a Daylight Savings Time/Summer Time zone when DST was not active, and vice versa.

Bug Fix

• Fixed client side binding for complex types containing floats or integers that was broken in version 0.7.5. Closes https://github.com/ClickHouse/clickhouse-connect/issues/335.

Improvement

• Added a raw_stream method to the Client the returns an io.Base. Use this instead of the raw_query method with the (now removed) optional stream keyword boolean. Thanks to Martijn Thé for the PR that highlighted the somewhat messy public API.

Bug Fix

• Fixed issue with SQLAlchemy Point type. Closes https://github.com/ClickHouse/clickhouse-connect/issues/332.

Bug Fixes

• Fixed client side binding for Python format strings using %d (int) and %f (float) format patterns. Closes https://github.com/ClickHouse/clickhouse-connect/issues/327
• Allows empty data argument in the initializer of ExternalFile / ExternalData objects. Thanks to martijnthe for the PR!

Improvement

• Added the new client method query_arrow_stream for streaming PyArrow queries from ClickHouse. Big thanks to NotSimone for the feature and tests! Closes https://github.com/ClickHouse/clickhouse-connect/issues/155.

Improvement

• Add summary field to Cursor object to retrieve the result of 'X-Clickhouse-Summary' header. Thanks to elchyn-cheliabiyeu for the PR!

Bug Fixes

• Inserts into columns with multibyte UTF-8 names were broken. This has been fixed. https://github.com/ClickHouse/clickhouse-connect/issues/312
• If the result of applying the precedence of timezones to a column results in an explicit UTC timezone, the datetime object returned should now be timezone naive. This should make the behavior consistent with the documentation. Closes https://github.com/ClickHouse/clickhouse-connect/issues/308 (except for a documentation update)
• Extraneous semicolons are automatically removed from the end of queries. Addresses the most basic behavior in https://github.com/ClickHouse/clickhouse-connect/issues/310.

Performance Improvement

• Pandas DataFrame returned from the client query_df method should be constructed somewhat faster in cases where the data returned in ClickHouse is in many small blocks. Note that performance gains in this use case are somewhat limited because of the memory and copying cost of building a large DataFrame from many smaller ClickHouse Native block structures, so such performance problems should normally be addressed at the query or ClickHouse data storage level (by for example, reducing the number of partitions and/or shards referenced by the query). This may partially address https://github.com/ClickHouse/clickhouse-connect/issues/307.

Bug Fixes

• Changed type hint of the query parameter in Client query* methods to Optional[str] to work correctly with type analyzers. This also highlights that using a query_context instead of a query in these methods is supported (and preferred for repeated queries). Thanks to Avery Fischer for the PR!
• Fixed sending a full table name to the insert_file tools function. Closes https://github.com/ClickHouse/clickhouse-connect/issues/305

Breaking Change

• Python 3.7 builds are no longer part of the wheels deployed to PyPI

Bug Fix

• Due to a change in default ClickHouse settings, inserts with "named" Tuple types no longer worked with ClickHouse version 24.1 and later. This has been fixed.

Improvements

• Some types of security and other proxies require additional query parameters on any call to ClickHouse server behind such a proxy. Because the HTTPClient makes certain initialization queries to ClickHouse before any query parameters are set, it was difficult or impossible to create a Client successfully. You can now modify the HTTPClient class level properties params and valid_transport_settings before calling get_client so that such "special" query parameters will be included even on initialization queries. Thanks to Aleksey Astafiev for highlighting the problem and contributing a PR.
• In some cases the user make want to disable urllib3 timeout settings connect_timeout and send_receive_timeout by setting them to none. The same PR from Aleksey Astafiev now allows setting to values to None
• Update to Cython 3.0.8

Bug Fix

• Add missing Nothing SQLAlchemy datatype, which fixes some edge case Superset queries. Thanks to elchyn-cheliabiyeu for the PR!

Improvement

• Avoid concatenation of empty dataframes during query_df due to Pandas future warning. Thanks to Dylan Modesitt for the PR!

Improvements

• Fix typo in log message for bad inserts. Thanks to Stas for the fix.
• Allow non ClickHouse Cloud tests to run on community Pull Requests
• Update to Cython 3.0.6

Bug Fix

• ATTACH queries were not be correctly processed as "commands". Thanks to Aleksei Palshin for the PR!

Improvements

• Added support for Point type. Closes https://github.com/ClickHouse/clickhouse-connect/issues/151. Thanks to Dhruvit Maniya for the PR!
• Upgraded to Cython 3.0.5
• Change exception handling in C API to stop spamming stderr

Bug Fix

• Fixed an issue where client side binding of datetimes with timezones would produce the incorrect time string if timezones differed between the client and ClickHouse server. Closes https://github.com/ClickHouse/clickhouse-connect/issues/268

Bug Fixes

• In some circumstances it was possible to insert a None value into a non-Nullable String column. As this could mask invalid input data, any attempt to insert None into a non-Nullable String or LowCardinality(String) will now throw a DataError
• Reading a named Tuple column where the Tuple element names contained spaces would fail. In particular this would cause expected failures reading the experimental JSON column type with spaces in the keys. This has been fixed. Closes https://github.com/ClickHouse/clickhouse-connect/issues/265. Note that handling spaces in column types is tricky and fragile in several respects, so the best approach remains to use simple column names without spaces.

Bug Fixes

• Reduce the estimated insert block size from 16-32MB to 1-2MB for large inserts. The large data transfers could cause "write timeout" errors in the Python code or "empty query" responses from ClickHouse over HTTPS connections. Should fix https://github.com/ClickHouse/clickhouse-connect/issues/258
• Ensure that the internal client _progress_interval is positive even if a very small send_receive_timeout value is specified. Closes https://github.com/ClickHouse/clickhouse-connect/issues/259. Note that a very short send_receive_timeout is not recommended.

Bug Fix

• Fix "negative" Date32 (before 1970-01-01) values for numpy and Pandas queries. Closes https://github.com/ClickHouse/clickhouse-connect/issues/254

Bug Fix

• Remove bad private import to fix C Linkage. Closes https://github.com/ClickHouse/clickhouse-connect/issues/252

Improvement

• Added Python 3.12 wheels and CI tests. Note that PyArrow is not yet available for 3.12, but should be soon. See https://github.com/apache/arrow/issues/37880
• The main clickhouse-connect.get_client method now displays type hints and ignores non-keyword arguments. Thanks to Avery Fischer for the usability improvement!
• Log messages regarding C optimization availability and JSON library selection have been change from INFO to DEBUG. Closes https://github.com/ClickHouse/clickhouse-connect/issues/249

Bug Fixes

• Fixed insert error when inserting a zero length string into a FixedString column. Closes https://github.com/ClickHouse/clickhouse-connect/issues/244
• Removed unnecessary validate_entrypoints import from top level package init that was breaking Python 3.7. Note that Python 3.7 is EOL and will no longer be supported as of January 1, 2024.

Bug Fix

• Fixed an issue with the automatic retry of "connection reset errors". This should prevent exceptions when the ClickHouse server closes a Keep Alive connection while a new request is in flight.

Improvement

• Improved support for typing tools by adding a py.typed file. Thanks to Avery Fischer for the contribution.

Bug Fix

• Nested empty Maps would return an IndexError when queried. https://github.com/ClickHouse/clickhouse-connect/issues/239. Thanks to Ashton Hudson for the report and the fix

Bug fixes

• Inserts using Pandas 2.1 would fail due to a removed method in the Pandas library. There is now a workaround/fix for this. Closes https://github.com/ClickHouse/clickhouse-connect/issues/234
• Inserts into a FixedString column that were not the expected size could cause corrupt insert blocks and mysterious errors from the ClickHouse server. Validation has been added so that more meaningful error messages are generated if a fixed string value is an invalid size. A reminder that strings which are "too short" for a FixedString column will be padded with 0 bytes, while strings that are "too long" will generate an exception during the insert.

Improvement

• Add support and tests for the Object(Nullable('json')) type, which is sometimes detected by schema inference.

Improvements

• Logging and exception handling for failed insert transformations has been reworked. If an exception is thrown when attempting to convert Python, Pandas, or Numpy data into ClickHouse Native format, the column name and type will be logged, as well as a stack trace of actual exception (note this may be in the C/Cython code, so the exception data may still be difficult to interpret). This partially addresses https://github.com/ClickHouse/clickhouse-connect/issues/229. Unfortunately determining data errors on a row level in addition to the column level is not practical in most cases without seriously impacting performance.
• Version information has been moved from a top level VERSION to a Python __version__ file in the package. This removes the Python 3.7 dependency on importlib_metadata.
• Cython .pyx, and .pxd files are now included in the PyPI source distribution to improve compatibility with 3rd party build tools.

Bug Fix

• Fixed client raw_insert method when a compression method specified. https://github.com/ClickHouse/clickhouse-connect/issues/223

Improvement

• Add compression parameter to the clickhouse tools.insert_file method. '.gz' and '.gzip' extensions are automatically recognized.

Bug Fixes

• Fixed an issue for older versions of ClickHouse where the server would send an initial block of 0 rows for larger queries. This would break some queries with LowCardinality columns. Closes https://github.com/ClickHouse/clickhouse-connect/issues/221
• Fixed thecompression alias for the compress client setting in SQLAlchemy/Superset DSN urls.

Improvements

• Upgraded to Cython 3.0.0 final release!
• Reversed the internal variable names of keys and indexes for low cardinality columns to be consistent with the ClickHouse server nomenclature.

Bug Fix

• Inserting into an Enum column from a Pandas DataFrame with integer values only inserted 0 values. This is fixed. https://github.com/ClickHouse/clickhouse-connect/issues/219

Bug Fixes

• The Client min_version method now ignores unrecognized "text" elements. This could cause issues for unofficial ClickHouse releases. Thanks to Diego Nieto for the fix!
• In most cases insert query is now sent as part of the POST body instead of as a query parameter. This fixes https://github.com/ClickHouse/clickhouse-connect/issues/213. Note that this does not happen for direct file inserts using the driver.tools module, since these rely on an unmodified buffered input stream to efficiently upload files. In that case the actual insert query will still be passed as a query parameter.
• All datetime objects returned from a query will now be timezone aware. This fixes https://github.com/ClickHouse/clickhouse-connect/issues/210. There remains one exception to this -- if the calculated timezone and the local timezone are both UTC, then naive timezones will be used to improve performance in such "all UTC" environments.
• Inserting Python dictionaries into a ClickHouse "named" Tuple column now works correctly. Fixes https://github.com/ClickHouse/clickhouse-connect/issues/215. Note that using dictionaries for inserts will be noticeably slower than inserting the equivalent Python tuple value (with elements in the correct order)

Improvements

• Client error messages used to be cut off at 240 characters to avoid creating huge log files. This value is now configurable using the common.max_error_size setting. Use 0 for this setting to get the full ClickHouse error message. In addition, the default has been changed to 1024 to capture more SQL errors without needing to modify the global setting value. Thanks to Ramlah Aziz for the update!
• All Client insert methods now return a simple QuerySummary object, which includes properties written_rows, written_bytes, and query_id calculated from ClickHouse HTTP response headers. A QuerySummary object is also returned from the Client command method if the command does not return other data. Closes https://github.com/ClickHouse/clickhouse-connect/issues/216
• Version determination no longer indirectly depends on the setuptools pkg_resources package. This also avoids some indirect dependency problems. Thanks to cwegener for the PR!

Bug Fixes

• Quote database name when retrieving tables via SQLAlchemy. Fixes the Superset issue https://github.com/apache/superset/issues/24372 for recent versions of Superset using clickhouse-connect
• Don't rely on the ClickHouse currentDatabase() function to set an explicit database parameter. This should not change functionality when no database is specified in Client creation since ClickHouse will use the user's default database in that situation regardless. Fixes https://github.com/ClickHouse/clickhouse-connect/issues/207

Bug Fix

• Inserts into decimal columns first convert the source value to a Python Decimal to work around floating point rounding issues. Fixes https://github.com/ClickHouse/clickhouse-connect/issues/203
• DateTime64 values were broken for dates before 01-01-1970. This is fixed. https://github.com/ClickHouse/clickhouse-connect/issues/204

Improvements

• Cython version upgraded to 3.0.0b3
• Inserts for string columns are now C optimized (approximately 2x faster)

Bug Fix

• Very long running queries could break because ClickHouse returned too many progress headers. Thanks to Ivan for the fix

Bug Fixes

• Use uuid4 instead of uuid1 for generating client level session_ids, as well as use a new urllib3 PoolManager when multiprocessing mode is detected. This should fix https://github.com/ClickHouse/clickhouse-connect/issues/194. Thanks to Guillaume Matheron for filing the issue and digging into details. The underlying problem is that the Python uuid1() is not guaranteed to be unique in a forked multiprocessing environment.
• Change log warning to debug message if numpy is not available for C bindings. This check is harmless if numpy is not installed and should not have produced a warning. Fixes https://github.com/ClickHouse/clickhouse-connect/issues/195

Improvements

• Cython version upgraded to 3.0.0b2
• The block size (number of rows) for chunked/streaming inserts is now dynamically determined based on sample of the insert data. This allows more efficient streaming of large inserts and significantly improves insert performance in some circumstances.
• Pivoting row based data to native columns for inserts has been optimized in C. This improves insert performance for large inserts of row oriented data.

Bug Fix

• The client will now validate that the client_protocol_version query parameter is actually received and used by the ClickHouse server before assuming that data returned confirms to the expected protocol version. This fixes an incompatibility with the current versions of CHProxy (and possibly other proxies that restrict the query parameters passed to the ClickHouse Server). Note that other features that require the use of query parameters (such as server side bound query parameters) may also fail because of this behavior in CHProxy. Fixes https://github.com/ClickHouse/clickhouse-connect/issues/191

Bug Fixes

• The client command method now accepts ClickHouse "external data." Closes https://github.com/ClickHouse/clickhouse-connect/issues/186
• Arrays of Python date and datetime objects are now correctly formatted when use as server side parameters. Fixes https://github.com/ClickHouse/clickhouse-connect/issues/188
• Fixed inserts of SimpleAggregateFunction columns with a LowCardinality type parameter. https://github.com/ClickHouse/clickhouse-connect/issues/187

Bug Fixes

• SQLAlchemy table reflection threw an exception for SimpleAggregateFunction columns. This has been fixed. https://github.com/ClickHouse/clickhouse-connect/issues/180
• The client no longer logs an invalid warning for query types that did not return a timezone header. https://github.com/ClickHouse/clickhouse-connect/issues/181
• Querying SimpleAggregateFunction columns with a LowCardinality type parameter was broken. This has been fixed. https://github.com/ClickHouse/clickhouse-connect/issues/182
• The query_arrow method now correctly accepts the external_data parameter. https://github.com/ClickHouse/clickhouse-connect/issues/183
• The query_arrow method has been fixed for read only queries/settings. https://github.com/ClickHouse/clickhouse-connect/issues/184

New Feature

• A common setting max_connection_age has been added, which will ensure that HTTP connections are not reused forever (this can help with certain load balancing issues. It defaults to 10 minutes

Bug Fix

• There was a critical issue when using zstd compression (the default) with urllib3 version 2.0+. This has been fixed.

Bug Fix

• Logging "Unexpected Http Driver Exception" only as WARNING instead of ERROR. Use the raised OperationalError if you depend on this. Thanks to Alexandro Sandre for the fix.
• The wait_end_of_query setting is no longer automatically sent with inserts. This caused unnecessary buffering on the ClickHouse server file system, especially in the case of many small inserts. It can still be added using the settings dictionary of the client *insert methods if needed for some reason.
• The query setting use_na_values has been renamed to use_extended_dtypes and now applies to all extended/special Pandas dtypes (except the Pandas Timestamp type). Set this to False to limit the dtypes returned in Pandas dataframes to the "basic" numpy types. (Note that this will force the use of numpy object arrays for most "nullable types") This should allow creating "basic" dataframes for greater compatibility. Closes https://github.com/ClickHouse/clickhouse-connect/issues/172.

Bug Fixes

• Fix Pandas dataframe inserts where the Dataframe index does not match the data values (after, for example, creating a new DataFrame from a subset of the original.) https://github.com/ClickHouse/clickhouse-connect/issues/167 Thanks to Georgi Peev for the report and suggested fix, and his continued stress testing of Pandas functionality.
• Compression and other control settings were not properly sent with the request if the corresponding setting was not enabled on the server. Many thanks to Alexander Khmelevskiy for the extended investigation and subsequent fix. https://github.com/ClickHouse/clickhouse-connect/issues/157

Bug Fixes

• Fix quoting and escaping of array literals in server parameters. See #159. Big thanks to Joachim Jablon for the report and the fix.
• Pandas and numpy Date values were incorrect for values after 2050. This has been fixed. https://github.com/ClickHouse/clickhouse-connect/issues/164
• Fixed server side parameter binding of the NULL value for Nullable types
• Added support for no_proxy/NO_PROXY environment variable. Also added support for lower case http_proxy and https_proxy variables. Note that lower case versions have precedence over upper case versions. Fixes https://github.com/ClickHouse/clickhouse-connect/issues/163

Performance Improvement

• The server timezone will not be applied (and Python datetime types will be timezone naive) if the client and server timezones match and the get_client apply_server_timezone parameter is True (the default). This improves performance where client and server have the same (non-UTC) timezone. To override this behavior and always apply a server timezone to the result, use apply_server_timezone='always'. This should fix https://github.com/ClickHouse/clickhouse-connect/issues/157

Timezone Improvements

• The client query_df and query_df_stream methods now accept query_tz and column_tzs parameters like other query* methods.
• A new boolean parameter apply_server_timezone has been added to the main get_client method. Setting this parameter to True (the default) will apply the server timezone (if not UTC) to values returned by the client query* methods. The previous behavior would always return timezone naive, UTC based Python and Pandas datetime objects for ClickHouse DateTime and DateTime64 columns without a defined timezone. To revert to the previous behavior, set the apply_server_timezone parameter to False. Closes https://github.com/ClickHouse/clickhouse-connect/issues/152
• The timezone logic applied to query results has been simplified and now uses the following order of precedence:
  • Use the column timezone for the column if it is specified using the column_tzs parameter
  • Use the column timezone for the column if specified in the ClickHouse column definition (only works for ClickHouse versions 23.2 and later)
  • Use the query timezone for the query if it is set using the query_tz parameter
  • Use the "response" timezone for the query as read from the X-ClickHouse-Timezone header if different from the server timezone. This closes https://github.com/ClickHouse/clickhouse-connect/issues/138.
  • Use the ClickHouse server timezone (if the client parameter apply_server_timezone is True)
• Note if the detected timezone according to the above precedence is UTC, clickhouse-connect will always return a naive datetime object with no timezone information

New Feature

• ClickHouse external data is now support for all client query methods. To send external data, construct a driver.external.ExternalData object and send it as the external_data parameter in the appropriate query method. See the ClickHouse documentation for additional details. There are also examples in the test file . Closes https://github.com/ClickHouse/clickhouse-connect/issues/98

Bug Fix

• Creating a client would fail if for some reason the user did not have access to the system.settings table. Thanks to Filipp Balakin for the fix.

Improvements

• String columns now accept values of bytes-like objects (bytes/bytearray/etc.) for inserts (as with other inserts, all values for the inserted column should be the same types, either a bytes-like object or str). A corresponding bytes read format has been enabled for String columns as well. Thanks to Tim Nooran for opening the issue and providing unit tests. https://github.com/ClickHouse/clickhouse-connect/issues/148
• Cython version upgraded to 3.0.0b1

Bug Fix

• Remove unnecessary addition of the client database to the table name for inserts. Fixes https://github.com/ClickHouse/clickhouse-connect/issues/145

Improvement

• The driver should now work for older versions of ClickHouse back to 19.16. Note that older versions are not officially tested or supported (like the main ClickHouse database, we officially support the last three monthly ClickHouse releases and the last two LTS ClickHouse releases). For versions prior to 19.17, you may want change the new readonly clickhouse_connect.common setting to '1' to allow sending ClickHouse settings with individual queries (if the user has write permissions). Thanks to Aleksey Astafiev for this contribution and for updating the tests to run with these legacy versions!

Bug Fix

• Remove direct pandas import that caused an unrecoverable error when pandas was not installed. https://github.com/ClickHouse/clickhouse-connect/issues/139

Improvements

• By default, reading Pandas Dataframes with query_df and query_df_stream now sets a new QueryContext property of use_pandas_na to True. When use_pandas_na is True, clickhouse_connect will attempt to use Pandas "missing" values, such as pandas.NaT and pandas.NA, for ClickHouse NULLs (in Nullable columns only), and use the associated extended Pandas dtype. Closes https://github.com/ClickHouse/clickhouse-connect/issues/132
• There are new low level optimizations for reading some Nullable columns, and writing Pandas dataframes

Bug Fixes

• Timezone information from ClickHouse DateTime columns with a timezone was lost. There was a workaround implemented for this issue in v0.5.8 that allowed assigned timezones to the query or columns on the client side. ClickHouse now support sending this timezone data with the column, but only in server versions 23.2 and later. If such a version is detected, clickhouse-connect will return timezone aware DateTime values without a workaround. Fixes https://github.com/ClickHouse/clickhouse-connect/issues/120
• For certain queries, an incorrect, non-zero "zero value" would be returned for queries where use_none was set to False. All NULL values are now properly converted.
• Timezone data was lost when a DateTime64 column with a timezone was converted to a Pandas DataFrame. This has been fixed. https://github.com/ClickHouse/clickhouse-connect/issues/136
• send_progress headers were not being correctly requested, which could result in unexpected timeouts for long-running queries. This has been fixed.

Improvement

• A new keyword parameter server_host_name is now recognized by the clickhouse_connect.get_client method. This identifies the "real" ClickHouse server hostname that should be used for HTTPS/TLS certificate validation, in cases where access to the server is through an ssh tunnel or other proxy with a different hostname. For examples of how to use the new parameter, see the updated file https://github.com/ClickHouse/clickhouse-connect/blob/main/examples/ssh_tunnels.py.

Bug fix

• The database element of a DSN was not recognized when present in the dsn parameter of clickhouse_connect.get_client. This has been fixed.

Bug Fix

• Referencing the QueryResult named_results property after other properties such as row_count would incorrectly raise a StreamClosedError. Thanks to Stas for the fix.

Improvement

• A better error message is returned when trying to read a "non-standard" DateTime64 column function for a numpy array or Pandas DataFrame. "non-standard" means a DateTime64 precision not conforming to seconds, milliseconds, microseconds, or nanoseconds (0, 3, 6, or 9 respectively). These DateTime64 types are not supported for numpy or Pandas because there is no corresponding standard numpy datetime64 type and conversion would be unacceptably slow (supported numpy types are datetime64[s], datetime64[ms], datetime64[us], and datetime64[ns]). A workaround is to cast the DateTime64 type to a supported type, i.e. SELECT toDateTime64(col_name, 3) for a millisecond column.
• The base configuration required for a urllib PoolManager has been broken out into its own help method, clickhouse_connect.driver.http_util.get_pool_manager_options. This makes it simpler to configure a SOCKSProxyManager as in the new example file https://github.com/ClickHouse/clickhouse-connect/blob/main/examples/ssh_tunnels.py

Improvement

• Reading Nullable(String) columns has been optimized and should be approximately 2x faster. (This does yet not include LowCardinality(Nullable(String)) columns.)
• Extraction of ClickHouse error messages included in the HTTP Response has been improved

Bug Fixes

• When reading native Python integer columns, the use_none=False query parameter would not be respected, and ClickHouse NULLS would be returned as None instead of 0. use_none=False should now work correctly for Nullable(Int) columns
• Starting with release 0.5.0, HTTP Connection pools were not always cleanly closed on exit. This has been fixed.

Bug Fixes

• Large query results using zstd compression incorrectly buffered all incoming data at the start of the query, consuming an excessive amount of memory. This has been fixed. https://github.com/ClickHouse/clickhouse-connect/issues/122 Big thanks to Denny Crane for his detailed investigation of the problem. Note that this affected large queries using the default compress=True client setting, as ClickHouse would prefer zstd compression in those cases.
• Fixed an issue where a small query_limit would break client initialization due to an incomplete read of the system.settings table. https://github.com/ClickHouse/clickhouse-connect/issues/123

Improvement

• Stream error handling has been improved so exceptions thrown while consuming a stream should be correctly propagated. This includes unexpected stream closures by the ClickHouse server. Errors inserted into the HTTP response by ClickHouse during a query should also be reported as part of a StreamFailureError

Bug Fix

• Return empty dataframe instead of empty list when no records returned from query_df method Fixes https://github.com/ClickHouse/clickhouse-connect/issues/118

Default parameter change

• The client query_limit now defaults to 0 (unlimited rows returned), since the previous default of 5000 was unintuitive and led to confusion when limited results were returned.

New Feature

• Allow client side control of datetime.datetime timezones for query results. The client query methods for native Python results now accept two new parameters: query_tz is the timezone to be assigned for any DateTime or DateTime64 objects in the results, while timezones can be set per column using the column_tzs dictionary of column names to timezones. See the test file for simple examples. This is a workaround for https://github.com/ClickHouse/clickhouse-connect/issues/120 and the underlying ClickHouse issue https://github.com/ClickHouse/ClickHouse/issues/40397 Note that this issue only affects DateTime columns, not DateTime64, although the query context parameters will override the returned DateTime64 timezone as well.

Bug Fix

• Http proxies did not work after removing the requests library. https://github.com/ClickHouse/clickhouse-connect/issues/114. This should be fixed. Note that socks proxies are still not supported directly, but can be added by creating a correctly configured urllib3 SOCKSProxyManager and using it as the pool_mgr argument to teh clickhouse_connect.create_client method.