Fixed

• Fixed dataframe_from_result_table crashing with pandas 3.0 when a datetime column contains all null values.
• Removed legacy pandas 1.x code path in parse_datetime (minimum supported pandas is now 2.3.1).

Fixed

• Managed Streaming Should Handle Throttling Events Correctly
• Changed extra's name back to aio
• Fixed handling of datetime columns in old pandas versions.
• Fixed encoding error in ingest_from_dataframe when using csv data format.

Changed

• [BREAKING] Minimum supported Python version is now 3.9, to align with other Azure SDKs.
• The project's infrastructure has been updated:
  • uv is used to manage packages
  • ruff is used for linting and formatting
  • basedpyright is used for type checking
  • poe is used to manage tasks
  • pdoc is used to generate documentation

Added

• Support Delos and Gov SG clouds

Added

• More endpoints supported by default.

Changed

• CloudSettings will now reuse the HTTP session from the query client for sync clients.

Fixed

• CloudSettings now cached by authority (schema, host and port) instead of full URL

Fixed

• Fixed Ingest from dataframe
• Dependency issues
• Typing issues

Fixed

• Fixed bug in ingest_from_dataframe with some ingestion properties combinations.

Fixed

• Dependency issues

Changed

• [BREAKING] Dropped support for python version 3.7, as it is on EOL for over year.
• [BREAKING] Aligned the Connection String Builder keywords with the rest of the SDKs. This means that some keywords were removed, and they will no longer be parsed as part of the Connection String.
  Building the Connection String using the builder method will still work as expected.
  The following keywords have been removed:
  • msi_auth / msi_authentication
  • msi_params / msi_type
  • interactive_login
  • az_cli
• [BREAKING] ingest_from_dataframe - Added data_format parameter. It can be None (default), 'json' or 'csv'. Based on how panda's csv serialization works, dynamic data will not be serialized correctly. By default, the data will be serialized as json to avoid this issue. However, that may break if a CSV mapping is used. Therefore, when passing the None value, the data will be json by default, or csv if a csv mapping is used. Also, it is possible to pass 'csv' or 'json' to force the serialization type.

• Explicitly export members in __init__.py via __all__

Fixed

• Added py.typed markers
• Fixed semantic error handling

Added

• Support more kusto endpoints by default

Fixed

• Better python errors when getting 401 and other http errors

Fixed

• Fixed bug in dataframe_from_result_table with some command results.

Added

• dataframe_from_result_table now accepts optional converters for specific columns or types.

Fixed

• Compatibility with numpy 2.0

Fixed

• Fixed bug in ManagedIdentity close method
• Fixed bug in IngestionResult repr

Changed

• Changed binary files data format compression to false

Fixed

• Removed test folders from wheel

Added

• Streaming ingestion from blob

Fixed

• Managed streaming fallback to queued
• Fixed token providers not being closed properly
• Internal test fixes
• Pandas support the new string type, if available

Fixed

• Pandas now correctly parses all dates

Added

• Added smart retry ability for queued ingestion.
• Support new playfab domain
• Added no-authenticaion option

Fixed

• Santaize secrets from blob urls
• Correctly handle closing of token providers in async (NOTE: now using async providers in sync clients will correctly raise an exception)
• Fixed proxy not passing correctly in some cases
• Fixed exception handling in web requests
• Internal fixes for environment variables
• Fixed documentation on E2E tests, and made it possible to test on a clean cluster

Changed

• Urls with one item after the path (i.e https://test.com/abc) will now be treated as cluster and initial catalog (ie. the cluster is "https://test.com" and the initial catalog is "abc").

Added

• Added Initial Catalog (Default Database) parameter to ConnectionStringBuilder
• Added callback parameter to device code
• Added method to manually set the cache for CloudSettings

Fixed

• Some edge cases in url parsing
• IgnoreFirstRecord now works properly
• Internal code improvement for telemetry

Fixed

• Unicode headers are now espaced using '?', to align with the service

Added

• Add new trident endpoint support

Security

• Redirects are no longer allowed

What's Changed

• Fixed resource parsing to be inline with other SDKs
• More resources are properly closed
• Internal testing fixes

Full Changelog: https://github.com/Azure/azure-kusto-python/compare/v4.1.1...v4.1.2

What's Changed

• Fixed context not awaited in traces, caused warning
• Fixed setup.py file, #453

Full Changelog: https://github.com/Azure/azure-kusto-python/compare/v4.1.0...v4.1.1

What's Changed

• Tracing support for the SDK
  • Use opentelemetry to trace the flow of operations when querying or ingesting
  • See quick_start for a usage example
• Add support for a TokenProvider to authenticate access.
  • KustoConnectionStringBuilder.with_token_provider() /KustoConnectionStringBuilder.with_async_token_provider()
  • Fixes #406 (DefaultAzureCredential can be provided)
• FIX - Use proper np types instead of strings, fixes #447
• FIX - samples file was cut short
• INTERNAL - Send http headers under the new unified format
• TIDY - Adding Microsoft SECURITY.MD

Full Changelog: https://github.com/Azure/azure-kusto-python/compare/v4.0.2...v4.1.0

What's Changed

• Add trident endpoint support
• Fixes #439 - Added more serialization options for "msi authentication" and "msi params" correctly
• Make error parsing less restrictive to support more errors

Full Changelog: https://github.com/Azure/azure-kusto-python/compare/v4.0.1...v4.0.2

What's Changed

• Added flag to handle nullable bools in pandas, fixes #436.
• Fixed source release not including a file.

Full Changelog: https://github.com/Azure/azure-kusto-python/compare/v4.0.0...v4.0.1

What's Changed

• [BREAKING] Trusted endpoints validation - by default, kusto will only connect to known endpoints.
• Added closability to the clients
• Make numpy dependency optional by @spektom
• Sample app improvements by @ronmonetaMicro
• Added Ignorefirstrecord ingestion property
• Fixed AttributeError when unpickling HTTPAdapterWithSocketOptions.
• Enum serialization fix
• Internal and testing improvements by @enmoed

New Contributors

• @spektom made their first contribution in https://github.com/Azure/azure-kusto-python/pull/402
• @enmoed made their first contribution in https://github.com/Azure/azure-kusto-python/pull/401
• @ronmonetaMicro made their first contribution in https://github.com/Azure/azure-kusto-python/pull/403

Full Changelog: https://github.com/Azure/azure-kusto-python/compare/v3.1.3...v3.2.0

What's Changed

• Allow passing service url with port
• Expanded mappings to support all options for all data formats
• Use "organizations" as the default tenant to support more cases
• Docs improvements by @nerd2ninja in https://github.com/Azure/azure-kusto-python/pull/393
• Internal refactoring

New Contributors

• @nerd2ninja made their first contribution in https://github.com/Azure/azure-kusto-python/pull/393

Full Changelog: https://github.com/Azure/azure-kusto-python/compare/v3.1.2...v3.1.3

What's Changed

• Fix bug when cluster URI ends with a slash '/' sign by @yogilad in https://github.com/Azure/azure-kusto-python/pull/392

Full Changelog: https://github.com/Azure/azure-kusto-python/compare/v3.1.1...v3.1.2

What's Changed

• Limit service client operations to 10 minutes by @AsafMah in https://github.com/Azure/azure-kusto-python/pull/384
• Add retry mechanism for potentially throttled processes. by @alonadam in https://github.com/Azure/azure-kusto-python/pull/381
• Added error recovery when parsing oneapi errors by @AsafMah in https://github.com/Azure/azure-kusto-python/pull/387

New Contributors

• @alonadam made their first contribution in https://github.com/Azure/azure-kusto-python/pull/381

Full Changelog: https://github.com/Azure/azure-kusto-python/compare/v3.1.0...v3.1.1

Features

• Added proxy support for all ingest clients, using the set_proxy method

Improvements

• All ingestion types (including JSON and AVRO) no longer require a mapping to work. In case a mapping is not provided, kusto will infer one based on the fields.
• dataframe_from_result_table will now save space by using int32 as the type for ADX's int

Changes

• DataFormat is now directly exported from azure.kusto.data, and is recommended to be imported this way.

This is a major release. It brings many new features, but also includes breaking changes.

Breaking Changes

• The minimum supported Python version is now Python 3.7. Python 3.5's EOL was a year ago, so you should upgrade regardless of this release.
• Exception messages may have changed. Make sure to test it if you're reliant on the exact message.
• Stricter checks on formats and mapping types. If your code breaks due to this, you had a bug.
• Renamed IngestionMappingType to IngestionMappingKind to align with other SDKs.
• Renamed ingestion_mapping parameter to column_mappings for clarity and alignment.
• KustoMissingMappingReferenceError and KustoMappingAndMappingReferenceError are now KustoMissingMappingError.

Data Format

• DataFormat is now exported directly from azure-kusto-data, and no longer exported from azure-kusto-ingest (use azure-kusto-data).
• DataFormat's internal representation changed from a string to an enum, including the Kusto service's formatting, whether a mapping is required, relevant IngestionMappingKind, and whether it's compressible.
• Added missing formats and mappings (SStream, ApacheAvro).

Major Features

Quickstart App

• Inside the repo there is now a new module for a Quickstart app. It serves as a sample and a tutorial for the entire flow for working with the SDK, including creating tables, ingesting data and querying.

Streaming Query

• A new API was added to KustoClient (sync and async) - execute_streaming_query.
• The regular execute call waits for all the data to arrive, parse it in memory, and provide Python objects to access it.
• Streaming query lets you work with the data as it arrives, saving time and memory.
• For usage, see the streaming query section in azure-kusto-data/tests/sample.py.

Managed Streaming Ingestion

• Adds a new type of ingestion client - ManagedStreamingIngestClient.
• The client will default to using streaming ingestion, but will be more robust to errors.
• If it fails to use streaming ingestion, it will fall back to queued when:
  • Retrying streaming ingest fails multiple times, getting transient errors
  • The size of the ingested object is more than 4MB
  • Using ingest_from_blob
• For usage, see the managed streaming section in azure-kusto-ingest/tests/sample.py.

Ingest Clients Changes

• All of the ingestion clients now inherit from BaseIngestClient.
• All of the ingestion clients now support the following methods - ingest_from_file, ingest_from_stream, ingest_from_dataframe (only Queued and ManagedStreaming support ingest_from_blob).
• IngestionResult is now returned from ingest_* methods. The result contains some details about the ingestion.
• For ManagedStreamingIngestClient, you can query whether the data was ingested via queued or streaming using IngestionStatus (== QUEUED for queued, == SUCCESS for streaming).

More Detailed Exceptions

• KustoServiceError now has two new subclasses - KustoApiError, KustoMultiApiError.
  • They both utilize the new OneApiError class, which has detailed information about the exception, for a consistent structure between SDKs.
  • KustoApiError is raised on a server error in the web request, and contains within it 1 OneApiError.
  • KustoMultiApiError is raised while reading the data, and can contain multiple OneApiErrors within.
• KustoClientError has a new subclass KustoBlobError, for when uploading a blob fails.
• Since these classes are subclasses of exceptions that already exist, they will not break existing code unless it parses and relies on the exact message.

Proxy

• Kusto Client now supports connecting with a proxy using the set_proxy(proxy_url: str) method.

Minor Features and Bug Fixes

• More type annotations in the code, to help provide completions and error detections.
• Binary types in ingestions will no longer be compressed.
• Providing the size for blobs for ingestion is now optional.
• Dataframe - support numpy types, including int, long, real and decimal.
• Dataframe - handle invalid date types.
• Relax Azure Identity version to support more dependencies.
• The client now sends a specific API version to avoid conflicts.
• Send correct scopes to managed identity and AZ CLI providers.
• Only retrieve CloudInfo when needed.
• Source IDs are always generated, and are always the correct type for improved telemetry.
• Added advanced checks for column mappings.
• Improved E2E test performance by only clearing cache once.

What's Changed

• Removed NoReturn reference, Solves #340

Improvements:

• Widened supported msal versions (>=1.9.0, <2) and azure-identity vesions (>=1.5.0,<1.7.0). Closes #333 #336
• Widened supported asyncio (>=3.4.4,<4) and asgrief (>=3.2.3,<4) versions for the aio option. Closes #337

Fixes:

• Async resources will now never be created for sync clients. That solves issues with creating clients in enviornments that don't have event loops, such as new threads. Fixes #332

Features:

• Added support for async callback to retrieve a token. use KustoConnectionStringBuilder.with_async_token_provider

Fixes:

• Locks are now local to an instance in TokenProviderBase. This should fix event-loop related issues. (#329)

Fixes:

• Fixed possible race condition when running multiple async queries at once

Improvements:

• Kusto now queries the provided cluster for metadata. This fixes incompatibility issues with other clouds, or services like Synapse. The results are cached and don't require any changes to the user's code or behavior. master

• Fixed status addressing between async and sync (#326)

Fixes:

• Return awaitable object from aio client aexit
• Fix ingestion endpoint format

Improvements:

• Use Azure.Identity AzCliTokenProvider to get a token from AzCli

Fixes:

• Fixed imports when aiohttp isn't installed. (Fixes #312)

Features:

• Add async io options using aiohttp to azure-kusto-data. This feature enables asynchronous calls that work with the built-in python coroutines Use the aio extra to include the client, and then access it via the azure.kusto.data.aio namespace.
• Add an option to login interactively with MSAL

Fixes:

• Handle possibility of unidentifiable OS or HTTPConnection.default_socket_options being None
• Fix MSAL default scope
• Fix helpers.to_pandas_timedelta conversion

The main change in this release is using MSAL instead of ADAL as an authentication library.

BREAKING CHANGES:

• Replace ADAL authentication library with MSAL.
• SDK dependencies have changed.
• Legacy IngestionProperties no longer usable.
• The KustoIngestClient class is now more accurately named QueuedIngestClient.

FEATURES:

• Support norequesttimeout ClientRequestProperty by setting the timeout to its maximum.
• Expose a retry parameter in the KustoClient init function.
• Improve calls to execute query performance by processing the result in a lazy manner.
• Expose raw query result to improve conversion into Pandas DataFrame.
• Add support of w3clogfile format.

FIXES:

• Send TCP Keep-Alive at regular intervals so that connections for long-running queries aren't dropped by the Load Balancer Service.
• Deletion of temporary files in case of an exception.
• Add client-server delta to all request timeouts.
• Add friendly error message when authentication with az_cli_auth and refresh token was expired.

BREAKING CHANGES:

• None.

FEATURES:

• Wrong endpoint detection - If the ingest client is constructed with a non-DataManagement endpoint, the user will receive a KustoClientError with a suggestion of the most likely DataManagement endpoint, instead of a generic KustoServiceError.

FIXES:

• Support independent cloud endpoint URLs.
• Fix parsing decimal values in KustoResultRow when using pandas DataFrame.
• Explicit handling of 404 errors.
• Fix an issue where the lack of payload in the response message caused an exception to be thrown.

OTHERS:

• Clean up data (drop created table) after running E2E tests.

FIXES:

• Bug: SNI auth option is ignored
• Bug: Token Provider auth is missing Federated Security flag

FIXES:

• MSI auth throws exception due to change in token result format

This release include some miscellaneous changes and features.

BREAKING CHANGES:

• Authentication via Managed Service Identity uses azure.identity instead of msrestazure. This library caches tokens which would resolve some issues for high capacity users. At the moment however, it only supports obtaining user managed service identities via the client_id hint. Attempting to provide KustoConnectionStringBuilder with an object_id or msi_res_id will result in a ValueError exception being thrown.

FEATURES:

• Added an option to authenticate using a certificate subject name and issuer. KustoConnectionStringBuilder.with_aad_application_certificate_sni_authentication(cluster, client_id, PEM, public_certificate, thumbprint, authority_id)

FIXES:

• Fix a bug where the exception text description reported to KustoServiceError were not saved due to a change in the syntax for explicitly calling a Super class constructor. Please mind that, -- The exception text description is saved in the args field of the Exception class. -- The message field no long exists in Python 3

• Raise descriptive exception when az cli token not found when using az cli authentication.

Release is mostly around changing the dependency for Azure Storage on the azure-kusto-ingest package.

It also includes various changes, which are mostly cosmetic, as well as adding E2E tests and adding them to the build pipeline as a check.

BREAKING CHANGES:

• Upgraded to use Azure Storage v12 (will require dependencies)
• All arguments changed to snake_case to conform to python guidelines.
• Renamed azure.kusto.data.request to azure.kusto.data.client - old import of from azure.kusto.data.request import KustoClient will now break. should change them to `rom azure.kusto.data import KustoClient

FEATURES:

None

FIXES:

• fixed breaking test (test_kusto_streaming_ingest_client.py)

OTHERS:

• Etoe tests - cleanup, grabbing connection parameters from env, and running inside buildpipeline.
• Added Contrib doc
• Added annotations where possible

BREAKING CHANGES:

• None

FEATURES:

• New option to authenticate by providing a function callback to KustoConnectionStringBuilder which returns a valid authentication token as a string, e.g.

token_provider = lambda: "{caller token}"
kcsb = KustoConnectionStringBuilder.with_token_provider("http://mykusto.kusto.windows.net", token_provider)

• When ingesting zip and gzip files, and uncompressed file size is not provided by the caller, attempt to detect uncompressed size from the file metadata.

• When ingesting, a user can provide inline ingestion mapping in the ingestion command instead of creating it in advance. We suggest to use the new columnMapping class instead of CsvColumnMapping or JsonColumnMapping.

FIXES:

• When ingesting a file and uncompressed file size is provided by the caller, use provided uncompressed file size instead of estimating the uncompressed size from the file size or file metadata

BREAKING CHANGES:

None

FEATURES:

None

FIXES:

• Minor fix for MSI auth

BREAKING CHANGES:

• no more py2 (reached EOL)

FEATURES:

• removed six - should reduce size when installing
• added better docs and cleaner py3 syntax where possible

FIXES:

• add az cli authentication: Once a user logs in to az cli using az login and then requests an access token using az account get-access-token, the client will be able to pick up the token and use it. client = KustoClient(KustoConnectionStringBuilder.with_az_cli_authentication('https://cluster'))

BREAKING CHANGES:

None.

FEATURES:

None.

FIXES:

• fixed bug described in https://github.com/Azure/azure-kusto-python/issues/150 when calling str() on KustoResultTable.

BREAKING CHANGES:

None.

FEATURES:

• Added support of authentication with Azure Managed Service Identity (MSI) via a new API KustoConnectionStringBuilder.with_aad_managed_service_identity_authentication(...) For more information about Azure MSI, see: https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview

FIXES:

• Prevent compression of binary formats (Avro, Parquet and Orc) when ingesting from local file.

BREAKING CHANGES:

None.

FEATURES:

Support ingestion of Orc format files

FIXES:

None.

BREAKING CHANGES:

None.

FEATURES:

None.

FIXES:

Fix parsing of streaming ingest service errors.

BREAKING CHANGES:

None.

FEATURES:

None.

FIXES:

• Locked Azure-Storage-* versions to 2.1.0.

BREAKING CHANGES:

None

FEATURES:

• If the use of a non-default AadAuthorityUri is needed, set environment variable AadAuthorityUri to the required uri.

FIXES:

None