azure-kusto-data

BREAKING CHANGES:

None.

FEATURES:

None.

FIXES:

• Before the fix, some imports were not performed when pandas is not installed, but were still referenced in code.

BREAKING CHANGES:

None.

FEATURES:

None.

FIXES:

• Before the fix, None values in non nullable column types (e.g Decimal) would raise when returned from queries. Now, will gracefully assign Nones Refer to test test_null_values_in_data for examples

BREAKING CHANGES:

None.

FEATURES:

• Added multijson support. this means that when trying to ingest json files that look like :

  [
      {...}, 
      {...}
  ]
  

  is now possible by specifying the format using ingestion properties.

FIXES:

None.

This release mostly contains minor improvements, most of which are to the KustoStreamingIngestClient

This is the first mention of it due to some minor alignments made.

In general, KustoStreamingIngestClient is meant for advanced use-cases, where you would want to ingest data immediately, bypassing the queued ingestion flow (KustoIngestClient). There are usage samples provided.

BREAKING CHANGES:

None.

FEATURES:

• KustoStreamingIngestClient minor improvemnets.
• Added the ability to pass an auth token with_aad_user_token_authentication
• Added __str__ to column mapping objects so that str(JsonColumnMapping()) will show useful debug info.

FIXES:

None.

BREAKING CHANGES:

• Change the type of server timeout from int/float into timedelta. So far, a code snippet as follow will run, but have the default timeout:

from azure.kusto.data.request import KustoClient, KustoConnectionStringBuilder, ClientRequestProperties
properties = ClientRequestProperties()
properties.set_option(properties.OptionServerTimeout, 8 * 60)
kcsb = KustoConnectionStringBuilder.with_aad_device_authentication(cluster)
client = KustoClient(kcsb)
response = client.execute(db, query, properties=properties)

Client request properties will now expect receiving timedelta. The following timeout will apply:

from datetime import timedelta
from azure.kusto.data.request import KustoClient, KustoConnectionStringBuilder, ClientRequestProperties
properties = ClientRequestProperties()
properties.set_option(properties.OptionServerTimeout, timedelta(8 * 60))
kcsb = KustoConnectionStringBuilder.with_aad_device_authentication(cluster)
client = KustoClient(kcsb)
response = client.execute(db, query, properties=properties)

FEATURES:

None.

FIXES:

None.

BREAKING CHANGES:

None.

FEATURES:

None.

FIXES:

• Fixed performance issues when converting from Kusto's table to pandas dataframe

BREAKING CHANGES:

None.

FEATURES:

None.

FIXES:

• Prevent messing up azure namespace on upgrade in environments with other azure packages.

---

More Info:

Old Kusto (<0.0.25) packaging (as documented by azure-sdk-for-python) included azure/__init__.py file which was the root of azure namespace and used pkg-resource to declare the namespace. After installation, Kusto's dist-info/RECORD included azure/__init__.py which caused it to be deleted once you upgrade (as upgrade is essentially uninstall + install newer). New setup excludes azure/__init__.py from packaging, because we declare a dependency on azure-nspkg which is the owner of the azure name space and is responsible for the site-packages dir. Because azure-nspkg is already installed, it's skipped, but the azure/__init__.py file is deleted (because it was present in the old dist-info/RECORD). There is a setup.py property called namespace_packages which is meant to be used with pkg-resource but, when specified, installing with it, prevents the removal of the namespace.

BREAKING CHANGES:

None.

FEATURES:

• Add the ability to set the source id of source descriptors. Useful for monitoring purposes (when reading statuses from queue, can correlate status via source_id).

import uuid
size = 1024 # insert correct size 
file_path = "/path/to/file"
source_id = uuid.uuid4()
file_desc = FileDescriptor(file_path, size, source_id)
ingest_client.ingest_from_file(file_desc , json_ingestion_props)

FIXES:

None.

BREAKING CHANGES:

None.

FEATURES:

None.

FIXES:

• errors working with proxy - fix reverts to use python requests which handles proxy env vars (HTTP_PROXY) properly

BREAKING CHANGES:

None.

FEATURES:

None.

FIXES:

• When getting empty KustoResultTable, let converting it into empty DataFrame. For more details please see the pull request comment.

BREAKING CHANGES:

• Change the default conversion from dynamic string.empty into dynamic null.

from azure.kusto.data.request import KustoClient, KustoConnectionStringBuilder
from azure.kusto.data.exceptions import KustoServiceError
from azure.kusto.data.helpers import dataframe_from_result_table

cluster = "https://help.kusto.windows.net:443"

kcsb = KustoConnectionStringBuilder.with_aad_device_authentication(cluster)
client = KustoClient(kcsb)
db = "Samples"

query = """print empty_dynamic = dynamic("")"""
response = client.execute(db, query)
empty = response.primary_results[0][0]["empty_dynamic"]

In the old version, the empty variable will have None type. From version 0.0.23 and on the empty variable will have "" value.

FEATURES:

None.

FIXES:

• Fixed json.loads exception in python 3.5 - Decode the urllib pool manager response with UTF-8, so we can load it into json in python 3.5 .
• Handling dynamic values which are not dicts/arrays- Removed unneeded json.loads call for dynamic objects. It's been loaded once after we got the response.

BREAKING CHANGES:

Stop supporting python 3.4 .

FEATURES:

• Support pandas 0.24.0 datetimes.
• Performance: Use connection pool instead of creating session for each request. For more detalis please read here.

FIXES:

• Raise better errors when trying to convert KustoResultDataTable into pandas DataFrame.

BREAKING CHANGES:

None.

FEATURES:

None.

FIXES:

• Azure packaging has changed. Kusto clients are now aligned. To read more about azure packaging, read this link.
• When converting datetime and timespan to pandas types, the conversion was not always correct. Read here to see how this was resolved.

BREAKING CHANGES:

• get_raw_response is no longer an option to: KustoClient.execute KustoClient.execute_query KustoClient.execute_mgmt The return type will be azure.kusto.data._response.KustoResponseDataSet

FEATURES:

None

FIXES:

None

Create ClientRequestProperties class.

BREAKING CHANGES:

• ClientRequestProperties class is introduced. The methods azure.kusto.data.request.KustoClient.execute* will no longer have the arguments accept_partial_results, timeout. In order to set them, an instance of ClientRequestProperties should be initiated and passed to the execute methods. Example:

from azure.kusto.data.request import ClientRequestProperties
properties = ClientRequestProperties()
properties.set_option(ClientRequestProperties.OptionDeferPartialQueryFailures, True)
properties.set_option(ClientRequestProperties.OptionServerTimeout, 10) # 10 seconds to timeout
r = client.execute_query("Database1", query, properties)

FEATURES:

• The ClientRequestProperties allow to pass options in the request body. See example above.

FIXES:

None

Add anonymous authentication.

BREAKING CHANGES:

None

FEATURES:

• KustoAuthenticationError is introduced. Once authentication fails, the exception will have details of the request to acquire the token.

FIXES:

• Fix issue https://github.com/Azure/azure-kusto-python/issues/73

Supporting dynamic types serialization both as dynamic (new serialization in query v2) and as string (old, for backwards compatibility).

BREAKING CHANGES: None

FEATURES: Support dynamic types in dataframe when dynamic types are serialized as dynamic (and not as strings).

FIXES: None

Add anonymous authentication.

BREAKING CHANGES:

• Default authentication method changed from device to anonymous . In order to use device login it should be specified explicitly KustoConnectionStringBuilder.with_aad_device_authentication.

FEATURES:

• Added Anonymous authentication option. You can send a request without an authorization header (default option when providing just a cluster uri KustoClient("https://<cluster>.kusto.windows.net.").

FIXES:

None

Mandatory authority_id for app key and secret authentication

BREAKING CHANGES:

• although it was technically mandatory to supply authority_id (tenant id) when authenticating with app key and secret, it is now a mandatory argument

FEATURES:

None

FIXES:

None

Fixing docs with regards to serializing kusto data table

BREAKING CHANGES:

None

FEATURES:

None

FIXES:

• Fix docs about how to serialize table + minor fix for KustoDataTable.str

Added the ability to track ingestion via status queues.

BREAKING CHANGES:

• ReportMethod.QueueAndTable is no longer a valid option. Currently only supported reporting method is Queues.

FEATURES:

• It is now possible to listen to queues for ingestion status. examples are provided here

FIXES: NONE

Pandas are no longer a requirement for the package. if pandas are needed, please install with optional flag (pip install azure-kusto-data[pandas] and pip install azure-kusto-ingest[pandas]).

BREAKING CHANGES:

• KustoResultTable.to_dataframe moved to helpers from azure.kusto.data.helpers import dataframe_from_result_table

FEATURES:

• it is now possible to work with your tenant id. exposed new authority_id as part of KustoConnectionStringBuilder. for further info on how to get the id - follow this link

FIXES:

• pandas are no longer installed as part of package which should make installation much faster.

Security: Add ability to authenticate with application certificate Interface: Kusto Connection String Builder is introduced. For more information look at: https://kusto.azurewebsites.net/docs/concepts/kusto_connection_strings.html

azure-kusto-data: Add refresh token mechanism. Make all 3 dimensions of Kusto response iterables: Response: iterate over tables. Table: iterate over rows. Rows: iterate over cells.

azure-kusto-ingest: Added resource manager.

General: Make the code black compatible.

Adding additional properties to ingestion properties. Changing requirements according to the use.

Add accept headers to rest calls. Add missing types while converting KustoResponse to DataFrame.

Fix adal dependency. Adjust ingestion data formats.