Features

• MaxStorage module - (EXPERIMENTAL) Introduces the new odps.maxstorage module, a pyodps-native flavor of the Storage API v2 support with a finalized end-user API surface. This adds full type annotations and ports blob support to the v2 Storage API including custom file names, nested blobs, and a BlobRecord dataclass. Blob uploads and reads are now streamed via chunked transfer-encoding instead of being materialized in memory, and download framing, compression defaults. A record-based API with automatic blob upload support is also added. The odps.apis.storage_api_v2 module is now deprecated and will be removed in a recent release.

Enhancements

• STS token support - Allows using the sts_token argument when constructing the ODPS entry, and fixes STS support for loading resources.
• HTTP proxy support - Merges environment variables to allow using HTTP_PROXY.
• Retry policy alignment - Aligns the retry policy with urllib3's error classification, and adds retry support for stream upload with slot reload on server errors.
• Stream truncation detection - Adds configurable require_stream_footer to detect stream truncation and avoid silent data loss in tunnel downloads. Also adds angle-bracket vector syntax (VECTOR<FLOAT,3>) test coverage.
• Code quality - Reduces bare exception handling To avoid potential lose of system errors like InternalServerError. Replaces mutable list closure hack with nonlocal declarations to meet Python 3 flavor.
• Variant stub - (EXPERIMENTAL) Adds a type stub for the variant type.

Bug Fixes

• Fix SQLAlchemy + MaxQA (MCQA) integration bugs - Resolves integration bug on quota names and result formats between SQLAlchemy and MaxQA (MCQA).
• Fix large_string/large_binary arrow cast overflow - Fixes overflow when casting to large_string/large_binary in the tunnel writer.
• Fix IndexError when writing to sparse block IDs - Fixes an IndexError that occurred when writing to sparse block IDs in the storage API.
• Fix concurrent race in CloudAccount v4 signature key cache - CloudAccount caches the v4 signing key keyed by date; concurrent sign_request calls could race the check-then-set on _last_signature_date / _last_signature_key.

Compatibility Notes

• odps.apis.storage_api_v2 deprecated - The odps.apis.storage_api_v2 module is now deprecated and will be removed in a recent release. Please migrate to odps.maxstorage, which provides the same Storage API v2 functionality with improved streaming, blob I/O, and nested-blob support.

Features

• Storage API v2 - Introduces the new Storage API v2 interface for interacting with MaxCompute storage.
• Vector type support - Adds full vector data type support including pure Python type definitions, tunnel I/O, and Arrow tunnel round-trip handling.
• BlobWriter and BlobReference for Python UDF - Provides stub classes for blob operations in Python UDFs.
• Free-threaded Python support (beta) - Introduces initial support for free-threaded Python (PEP 703), allowing PyODPS to run in environments without the GIL.

Enhancements

• Drop support of Python 2.7 - Support of Python 2.7 is dropped and compatibility modules are removed. Code is changed to accept new syntax of Python>=3.7.
• Try obtaining task statuses when task results are ready - Improves task status retrieval by attempting to obtain statuses when results become available. This can help obtaining results under MaxQA.
• Add missing tunnel session arguments - Adds previously missing arguments to tunnel session creation.
• Add fallbacks for enums - Adds fallback handling for enum values to improve robustness.

Bug Fixes

• Use struct for timestamps from instance tunnel - Improves timestamp handling in instance tunnel by using struct-based parsing.
• Fix potential exponential recursion in PyODPS DataFrame - Introduces caching for ExprProxy objects to avoid exponential recursion.
• Remove monkey patch of minidom writexml method - Fixes incompatibility of the patch in Python>=3.13.
• Fix writing upsert table with random primary key location - Fixes cython version of RecordHasher to produce correct hash keys.

Compatibility Notes

• Python 2.7 support dropped - PyODPS v0.13.0 no longer supports Python 2.7. The minimum supported Python version is now 3.7. Please do not upgrade if you want to stay with Python 2.7. DataWorks users who are using PyODPS with Python 2.7 will stay with PyODPS 0.12.6.
• Mars removed - Mars integration support has been removed. Users relying on this should switch to MaxFrame or migrate to alternative solutions before upgrading.

This release backports several fixes and features from the master branch to the 0.12.6 branch, introducing the Variant and Vector data types, improving account signing mechanisms and concurrency safety, and fixing a RecordHasher crash in specific scenarios.

Features

• Variant and Vector data type - Adds the Variant and Vector primitive data type, allowing these two types to be declared. Note that this only supports type definitions. You need to upgrade to later PyODPS versions on Python 3 to truly use these types.
• sts_token constructor argument - Supports passing sts_token directly when constructing the ODPS entry object, which automatically creates an StsAccount, simplifying STS scenario initialization.

Bug Fixes

• Fix RecordHasher IndexError when PK columns are not at the beginning of schema - The Cython version of RecordHasher would crash with an index out-of-bounds error when primary key columns were not located at the start of the schema. Replaced with an independent key_idx counter to map primary key columns, fixing the issue. (aliyun/aliyun-odps-python-sdk#324)
• Fix concurrent race in CloudAccount v4 signature key cache - CloudAccount caches the v4 signing key keyed by date; concurrent sign_request calls could race the check-then-set on _last_signature_date / _last_signature_key.
• Fix IndexError when writing to sparse block IDs - _open_writer stored sub-writers indexed by block-id value while the rest of AbstractTableWriter indexes _blocks_writers by position, breaking open_writer(blocks=[3, 7]) ... write(3, ...). Threaded the positional idx through _open_writer and asserted the block_id/idx invariant at the storage point, restoring the v0.9 positional contract.

Features

• Made pyodpswrapper available externally. When images of DataWorks are upgraded, users can upgrade features pyodpswrapper through pip install -U pyodps
• Added Python 3.13 and 3.14 wheel support (without free-threaded support yet)
• Added support for multi-thread read and write operations
• Added EnhanceWriteCheck parameter to the CreateWriteSession interface in the storage API
• Added partition_spec argument support for open_reader and open_writer methods

Enhancements

• Added SQL hints parsing for SET lines
• Allowed server-side fallback with MCQAv2
• Added caching mechanism for URL resources

Bug Fixes

• Fixed pickling of CredentialProviderAccount objects
• Fixed uploading timestamp types in pandas DataFrames
• Fixed writing empty columns with write_table function
• Fixed checks for quotaName parameter in tunnel operations
• Fixed long latency when submitting large SQL statements
• Avoided unescaping CDATA sections twice

Docs

• Fixed wrong raise information in ODPS.get_xxx methods

Bugfixes

• Fix pickling error of CredentialProviderAccount.

Enhancements

• Allow using ODPS_REGION_NAME to pass region name in envs.
• (Experimental) Add support for catalog API and model read methods.
• Support hash list of data for upsert.
• Add support for BufferedRecordReader to reduce connection period.
• Add on_exception handler to control retry.
• Add namespace param to tunnel_rest.
• (Experimental) Add stub for blob type. Reading and writing for the type not implemented yet.

Bugfixes

• Sync tzlocal workaround for linux tz.
• Fix precision check in decimals. (#292)

Compatibility issues

• Since v0.12.5 decimal precision and scale check at client side is made tighten to meet the behavior of MaxCompute at server side. Checks previously worked in earlier releases may break with the new release.

Notes

• Blob type is added as a stub to avoid errors in table creation and metadata handling. Data reading and writing is not supported yet.
• Models are added as an experimental feature and currently we only support getting and listing models.

Bugfixes

• Lock when updating and retrieving access ids & keys in certain accounts.
• Add schema name when retrieving temp table in write_sql_result_to_table.
• Allow SQL starts with WITH to be select stmt when calling MCQA.
• Pass inst_kw argument in more table functions invoking SQL.

Enhancements

• Add an option to allow casting to arrow dtypes on arrow tunnel.
• Allow creating table with complex types with write_table.
• Add an option infer_type_with_arrow to allow infering data type with pandas with write_table.
• Allow selecting record tunnel when calling to_pandas methods.
• Add support for binary types when using SQLAlchemy.
• Remove newsgroup data example due to security concerns about tarfile module.
• Raise an error when running SQL interactively with mcqa_v2=True and quota is not returned from server side (for instance, requesting on a legacy quota).
• Add kwargs arguments for table "SQL" methods to allow more options for instances.
• Add type stub for geography type.
• Fix image tags of python-pack to make sure Python 3.7 is still available.

Bugfixes

• Fix errors when special chars inside backquotes and backquotes within names.
• Keep in-statement comments when splitting SQL statements to avoid losing functional comments like join hints.
• Replace possibly malfunctioning UTC zone from ZoneInfo with datetime.utc if possible.
• Fix error when popping writer from cache when storing local DataFrames with DataFrame API. It is recommended to use write_table instead of DataFrame APIs if you only need to write your local pandas data to MaxCompute.
• Restrict numpy version to <2.3.0 to avoid installing numpy from source during installation.

Compatibility issues

• Since PyODPS 0.12.4, job-insight urls are used by default instead of logviews if job-insights are enabled at server side. Configure options.use_legacy_logview = True if you do not expect this behavior.
• Since PyODPS 0.12.4, odps.accounts.AliyunAccount is renamed as odps.account.CloudAccount to meet the requirements of certain region. Please change your imports.
• Since PyODPS 0.12.4, V4 signature is enabled by default. If your service does not support it, you can turn it off by configuring options.enable_v4_sign = False to avoid request attempts with the new signature.
• Since PyODPS 0.12.4, image tags of python-pack is fixed. To utilize existing local manylinux images you can add an argument --use-default-image-tag.

Notes

• Geography type is added as a stub to avoid errors in table creation and metadata handling. Data reading and writing is not supported yet.

Features

• Add support for resources for VolumeFile and VolumeArchive for external volume files.
• Implements method to write SQL result to table.
• [Experimental] Add support for auto-partition table.

Enhancements

• Ignore cases for column names in schema and record.
• Remove decimal precision & scale check to allow large decimal scale.
• Add config for logview latency and print final progress.
• Enhance DDL generation for ROW FORMAT SERDE clause.
• Add global compress options for tunnel.
• Add session refresh option for Storage API.
• Add unique identifier ID to instance job model.
• Automatically add default settings in xflow instances.
• Allow equality comparison of columns and records.
• Allow arrow instance tunnel to use multiple processes.
• Add anonymous placeholder for DBAPI SQL parameterized query.
• Allow specifying dict type for structs when options.struct_as_dict == True.
• Allow specifying table creation args and column types when calling write_table with create_table == True.
• Add raw error message to warning when PermissionError encountered in downloading data from instance tunnel.
• Automatically convert all settings to string in MaxFrame task to avoid unexpected behavior from server.
• Add cast for complex types to make sure arrow conversion returns correct result.
• Add create_partition argument for tunnel APIs.

Bugfixes

• Fix error when loading ODPS engine spec of superset.
• Fix cast for string column to new decimal type in arrow tunnel.
• Fix size check issue for varchar type with binary data.
• Fix error when None elements in complex types.
• Fix huge space between lines of repr(table) when name or type is long.
• Fix inferring table schema from arrow schema when struct type exists.

Documentations

• Add docs of table functions.
• Add docs for diagnosing slow SQL execution.
• Add docs of ODPS types.
• Add links to API references for object docs.
• Add API docs for tunnels and remove Mars docs.
• Mark PyODPS DataFrame as deprecated in documentation.

Compatibility issues

• Since PyODPS 0.12.3, column names with different capitalization are considered the same to make it compatible with behaviors of MaxCompute service. It may break some cases where capitalized column names are used.
• Since PyODPS 0.12.3, dict instead of OrderedDict is used by default for structs when options.struct_as_dict == True for Python>=3.7. Change to legacy behavior by setting options.struct_as_ordered_dict = True.

Bugfixes

• Fixes potential corruption of default global settings.

Enhancements

• Add session refresh option for storage API.
• Enable timeout when using asyncmode for table tunnel.
• Enhance DDL generation for ROW FORMAT SERDE clause.

Documentation

• Add docs for basic types, tunnel and table functions.

Bugfixes

• Fix error when loading ODPS engine spec of superset.

Features

• (Experimental) Add support for MCQAv2 for sqlalchemy.
• Add table alternation utility functions
• Add support of job insight instead of logviews. Can be turned on by configuring with options.use_legacy_logview = False.

Enhancements

• Make SuperSet support compatible with SuperSet 4.1.0 and later.
• Print usage when command not correct for pyodps-pack.
• Add support for timestamp_ntz for arrow tunnel.
• Add checks for potential None header values before request.
• Add retry when getting credentials from providers.
• Add retry when schema of stream tunnel mismatches with current schema.
• Make odps.task.wlm.quota available for MCQAv2 to set quota name.
• Stop using platform.platform to check OS type.
• Escape comment in DDL to allow quotes within comments.
• Enable using field readers and writers in C tunnel implementation.

Bugfixes

• Fix tags header for tunnel readers and writers.
• Fix duplicate param error in to_pandas method in partitions.

Documentation

• Add installation notices for urllib3 version when ssl module is compiled with openssl<1.1.1.

Bugfixes

• Add an import to requests in odps.lib to resolve compatibility issue of legacy codes.

Features

• (Experimental) Add metrics interface for tunnel.
• (Experimental) Add support for MCQAv2.
• Add support for upsert writer for table object.

Enhancements

• Support hashing of decimal types for primary keys.
• Shift CSV field size limit to table field size limit when reading with legacy result interface.
• Add cythonized decimal, array, map and struct validators to accelerate reading and writing of arrays.
• Add allow_schema_mismatch option and CDC info on tables and partitions.
• Enhance call_with_retry to support KeyboardInterrupt and ignoring exceptions.

Bugfixes

• Fix mishandling of project name passed to TableTunnel constructor when getting table by name only.
• Fix duplicate param error in to_pandas method in partitions.
• Fix re-obtaining bearer-token when it meets timeout.

Documentation

• Fix dead URL for the guide to run PyODPS DataFrame in cluster.
• Refine description of tunnel download limit.

Features

• Implements write_table with pandas to facilitate creating tables or partitions with pandas DataFrames.and to_pandas methods to facilitate converting from and to pandas DataFrames.
• Add support for converting table data and instance results to pandas DataFrames with to_pandas and iter_pandas methods.
• Add separate delete methods for views and materialized views.
• Add support for table freeze command.
• Add support for using computational quotas.
• Add params to allow creating and removing root directory of external volumes.
• Allow direct method to obtain SQL statement from instance objects.
• Supports using AlibabaCloud credentials to access MaxCompute.
• Add support for append_partitions argument on tunnel reader and writer.
• Support complex types in UDF debug utility pyou.
• Add support of seeking VCS directory roots with pyodps-pack.

Enhancements

• Allow record and reuse MCQA session with local file.
• Move session methods into models package.
• Enable black lint for repository and unify quotes with double quotes if possible.
• Optimize handling of arrow tunnel timezones with pyarrow.compute if possible.
• Add tags header for tunnel session requests.
• Allow creating ODPS entry via environment variables.
• Add definition of MaxFrame tasks.
• Split task module into multiple modules by task category.
• Move table download retry from table API into base tunnel.
• Enhance support of full resource paths and temp resources.
• Add default task_name for instance methods.
• Upgrade tblib to 3.0.0 and refine compatibility for Python 2.7.
• Allow caching object names for SQLAlchemy to accelerate table listing.
• Enable instance waiting at server side and add retrys for methods on instance object.
• Set default value of project_as_schema given tenant schema support in DBAPI support.
• Add retry and error logging when using multiprocessing to read.
• Remove support of legacy pandas IO support for simple types.
• Support schema version on stream upload session.
• Allow creating STS account from environment variables and force reload on expiration.
• Allow raising errors when task info resposes are empty by passing raise_empty=True.
• Allow creating tunnel sessions with full table name.
• Rename options.always_enable_schema as options.enable_schema to make it consistent with MaxFrame configutations.
• Switch TableTunnel.create_download_session to async_mode by default to reduce chances of failure when calling tunnel SDK directly.
• Warn when running pyodps-pack with sudo under macOS. Show warning and error messages with colors as well.

Bugfixes

• Fix error when uploading multiple batches and multiple blocks with BufferedArrowWriter.
• Fix field size error when it is specified in project settings.
• Fix odps.merge.txn.table.compact argument of merge compact command.
• Fix compatibility for Numpy 2.0.
• Upgrade six and setuptools requirements under Python 3.12 to fix installation issue.
• Rewrite do_ping errors when called by Apache Superset to fix misleading error messages.
• Fix compatibility of Apache Superset date functions.
• Fix usage of sqlalchemy with bearer token account.

Documentation

• Add more documentations for tunnel APIs.
• Normalize zh strings in po files by using jieba to split words.

Deployment & Tests

• Make tests more stable by using distinct table names and ordered dict.
• Refine tests to make it possible to test with pytest-xdist.
• Fold full requirements inside requirement strings (thanks @dimbleby).

Compatibility Issue

• Undocumented APIs open_pandas_reader and open_pandas_writer are removed from TableTunnel class to make wheels simple. Users who call these APIs should use to_pandas, write_table or arrow tunnel support instead.
• options.always_enable_schema is renamed as options.enable_schema and the former option is now marked as deprecated.
• --without-docker and --without-merge options in pyodps-pack are renamed as --no-docker and --no-merge to make them consistent with pip style.

Enhancements

• Switch TableTunnel.create_download_session to async_mode by default.
• Support schema version on stream upload session.
• Allow creating STS account from env and force reload on expiration.

Bugfixes

• Fix error when uploading multiple batches with BufferedArrowWriter.

Bugfixes

• Fix CRC computation of arrow tunnel interfaces
• Fix completeness of upload retry of buffered writers

Enhancements

• Allow record and reuse MCQA session with local file

Tests

• Fix test failure of storage API

Bugfixes

• Fix types support of json and timestamp_ntz in sqlalchemy
• Fix odps.merge.txn.table.compact argument of merge compact command
• Make compatibility for Numpy 2.0

Enhancements

• Warn when running pyodps-pack with sudo under macOS
• Allow reading envs from ODPS.init

Documentation

• Add more docs for tunnel APIs

Enhancements

• Allow opening resources with full resource path and temp hint
• Add MaxFrameTask to models

Features

• Add support for cluster info and views in tables and table DDL output.
• Add support for easier threaded writing and writing in multiple processes for TableWriter.

Enhancements

• Use monotonic time to calculate timeout.
• Add support for http+unix socket connection.
• Optimize RequestsIO by introducing buffering and simplify threaded sync.
• Revoke embedded requests and use buffered writer for table API by default.
• Add cython converter for legacy decimal.
• Store local configs inside context variables if possible.
• Add support for zoneinfo of Python standard library since Python 3.9.
• Show SQL statement when encountered ParseError.
• (experimental) Add support for new V4 signature. The signature is turned off by default.
• (experimental) Add support for accessing MaxCompute with AlibabaCloud credentials.
• (experimental) Upgrade six and setuptools requirements under Python 3.12.

Bugfixes

• Fix TypeError when calling open_resource to create resources.
• Fix superset error when reuse_odps=true.
• Fix errors when decimal digits are not sufficient.
• Fix support for DataFrame UDFs under Python 3.11.
• Fix timezone of arrow tunnel to make it consistent with record tunnels.
• Fix comparison of date sequences in DataFrame.

Tests

• Add check for malicious tunnel requests.

Documentation

• Fix default timezone document.
• Add detailed usage for reading table data under multiprocessing and multiple threads.
• Add detailed info for low-level tunnel interfaces.

Compatibility issues

• Timestamp objects obtained with arrow tunnel now uses local timezone instead of UTC to keep consistency with record tunnels. Please update your code when you've already done manual timezone conversion.

Bugfix

1. Fix attribute errors for table preview and storage API.

Features

• Add support for arrow table preview reader
• Enhance support for Apache Superset
• Add support for storage tier on tables and partitions
• (Experimental) Add support for tunnel upsert
• (Experimental) Add image argument for DataFrame

Bugfixes

• Fill partition value for tunnel records
• Use PERCENTILE_APPROX for doubles under ODPS 2.0
• Convert all requirement files to UNIX format for pyodps-pack
• Fix error when reloading volume tunnel session
• Fix logview setting not working in options
• Dump SQL statement when encountered ParseError
• Remove misplaced warnings when pickling user functions
• Fix errors of to_pandas for InSessionInstance readers
• Fix position of tablesample clause for sample
• Fix compatibility for SQLAlchemy 2.0
• Fix results of value_counts when values are None
• Remove empty equal mark for url actions
• Stop copying and caching for DataFrame(pd).persist if possible to reduce memory usage
• Fix missing quotaName in full lifecycle of tunnel requests
• Fill partition value for tunnel records
• Fix starting of Mars notebook and Mars import in some case
• Delete deflate Content-Encoding header for halo in storage api

Enhancements

• Supports scanning dependencies for pkg_resources
• Add PEP517 args for pyodps-pack
• Persist pandas dataframes in batches
• Use date in response headers to replace fields in Schemas
• Add detailed logs for sign server on errors
• Make option context as thread locals
• Adapt to extended types for ODPS arrow format
• Supports schema API along with SQL implementations
• Add support for MaxFieldSize passed by server end
• Add options to allow keeping resources for DataFrame
• Add support for timestamp_ntz type
• Refine error message for malfunctioning create instance response
• Allow adding custom log handlers to support displaying logs in notebook kernels
• Allow using run_sql to execute merge smallfiles or compact commands
• Allow specifying transactional table property
• Unify verbose_log into standard Python logging and dump progress when waiting for instances
• Return struct values as namedtuples by default and fix DataFrame customized functions on complex types
• Add retry for BufferedRecordWriter when writing blocks
• Reuse task utilities to simplify MCQA submission

Documentations

• Fix pyodps-pack doc on docker requirements
• Add doc for timezone setting
• Make bare tunnel docs more explicit
• Refine documents of instance tunnel limit

Compatibility issues

• PyODPS now returns struct values as namedtuples for tunnels to keep consistency with UDFs. For most of the cases your code might still work. If it doesn't, try configuring options.struct_as_dict = True.
• From v0.11.5 nullable property of columns is added for transactional tables, and default value for partition columns is False. If you use these column instances in some scenario, for instance, using them as common columns to create tables, non-nullable columns could be created and insertion of null values will result in errors. To ignore nullable flags in columns, try configuring sql.ignore_fields_not_null = True.

Bugfixes

• Stop copying and caching for DataFrame(pd).persist if possible to reduce memory usage.

Features

• Add support for arrow table preview reader
• Enhance support for Apache Superset
• Add support for storage tier on tables and partitions
• (Experimental) Add support for tunnel upsert

Bugfixes

• Fill partition value for tunnel records
• Use PERCENTILE_APPROX for doubles under ODPS 2.0
• Convert all requirement files to UNIX format for pyodps-pack
• Fix error when reloading volume tunnel session
• Fix logview setting not working in options
• Dump SQL statement when encountered ParseError
• Remove misplaced warnings when pickling user functions
• Fix errors of to_pandas for InSessionInstance readers
• Fix position of tablesample clause for sample
• Fix compatibility for SQLAlchemy 2.0
• Fix results of value_counts when values are None
• Remove empty equal mark for url actions

Enhancements

• Supports scanning dependencies for pkg_resources
• Add PEP517 args for pyodps-pack
• Persist pandas dataframes in batches
• Use date in response headers to replace fields in Schemas
• Add detailed logs for sign server on errors
• Make option context as thread locals
• Adapt to extended types for ODPS arrow format
• Supports schema API along with SQL implementations
• Add support for MaxFieldSize passed by server end
• Add options to allow keeping resources for DataFrame
• Add support for timestamp_ntz type
• Refine error message for malfunctioning create instance response
• Allow adding custom log handlers to support displaying logs in notebook kernels
• Allow using run_sql to execute merge smallfiles or compact commands
• Allow specifying transactional table property
• Unify verbose_log into standard Python logging and dump progress when waiting for instances
• Return struct values as namedtuples by default and fix DataFrame customized functions on complex types

Documentations

• Fix pyodps-pack doc on docker requirements
• Add doc for timezone setting
• Make bare tunnel docs more explicit
• Refine documents of instance tunnel limit

Compatibility issues

• PyODPS now returns struct values as namedtuples for tunnels to keep consistency with UDFs. For most of the cases your code might still work. If it doesn't, try configuring options.struct_as_dict = True.

Enhancements

• Reuse UDFs when code is same and without closures
• Add function to show versions of dependencies
• Make stream tunnel to write in blocks
• Add quota_name params for various tunnel sessions
• Refine MCQA execution API and fallback behavior
• Supports JSON column type
• Use TABLESAMPLE clause to implement sampling with frac or rows
• Allow packing dynamic libraries with pyodps-pack
• Auto resolve source dependencies in no docker mode in pyodps-pack

Bug fixes

• Fix jump targets when jump instruction size changes
• Fix auto-flush for arrow writers

Deployment

• Restrict urllib3 version to 1.x.

Features

• Add API-by-API implementation for storage API
• Add retry for table read API
• Add automatic submission for table write API

Bugfixes

• Fix OSError caused by BPO-29097 under certain Python versions
• Show composite error message when failed to parse data type

Enhancements

• Drop support for Python 2.6
• Add more options of pip into pyodps-pack
• Show more information when command not found on pyodps-pack
• Refine creating ODPS instances from environment variables
• Use modified requests library to simplify file-like writers
• Optimize cython implementation of tunnel record IO by introducing more nogil marks
• Refine error parsing and add tag of endpoint
• Reduce calls of tenant APIs
• Add options to read antique datetime as None
• Add supports for minikube for pyodps-pack
• Support yielding data while writing in arrow tunnels
• Support to_pandas on slices of readers

Deployment

• Fix dir missing on installing with source code with Jupyter

Tests

• Migrate all tests to pytest

Documentation

• Require jQuery for documentations
• Add notifications for checking XFlow instances aster iter_xflow_subinstances.

Compatibility Issues

• Supports of Python 2.6 is formally dropped since 0.11.4. Please use 0.11.3.1 or earlier versions.
• Using async_ arguments as position arguments is deprecated. Please use it as a keyword argument.
• BufferredRecordWriter is now renamed as BufferedRecordWriter. References to old class should be switched into new one.

Enhancements

• Add support for none-Docker mode for pyodps-pack. It now supports limited scenarios when Docker not available.
• Reduce maximum memory cost of to_pandas() on tunnels by converting to pandas in batches
• Supports complex types when calling to_pandas() on tunnels
• Use default schema when odps.namespace.schema enabled on tenants, or options.always_enable_schema set to True
• Make sure merging small files is available under schemas
• (Experimental) Supports more functionality of external volumes

Bugfixes

• Fixes tunnel writing when pd.NA is used

Documentation

• Multiple documentation fixes

Features

• Add new command line tool pyodps-pack to pack third-party libraries, recommended as standard packing mechanism
• (Experimental) Add preliminary support for custom DataFrame functions with Python 3.8 / 3.9 / 3.10
• Supports DataFrame column join methods
• Support configuring instance settings via connection strings with SQLAlchemy
• (Experimental) Supports external volume
• Supports run_sql_interactive_with_fallback interface in pyodps
• Supports get_max_partition for tables
• Implements stream read and write of resources
• Supports iterationg table partitions with logical conditions

Enhancements

• Supports quota name when creating tunnels
• Allow apply UDFs to use __getitem__
• Reset table object when alter table is called
• Allow persisting DataFrames with table objects
• Support compression with LZ4 and zstd for tunnels
• Remove dependencies of deprecated distutils package
• Rename last_modified_time with last_data_modified_time for clarity
• Refine error display and arrow format when fetching data with multiprocessing

Bug fixes

• Fix downloading with multiprocessing under Windows
• Fix negative timestamp issue under Windows
• Fix multiple issues on Github
• Resolve coded json when parsing
• Add retry when encountering MetaTransactionFailed
• Fix parsing RFC822 date when loading ODPS meta
• Fix schema error when calling to_pandas() with columns specified
• Fix handling NaNs with fillna
• Fix compiling SQL scalars with different integer types

Compatibility Issues

• As new Schema object level is introduced, it is now discouraged to use schema for table schemas and warnings will be produced. Try using table_schema instead when you code with the new version.
• Attributes like creation_time now uses local time instead of UTC time for consistency with other datetime attributes by default. Switch to old behavior by setting options.use_legacy_parsedate = True.
• Attribute last_modified_time on tables and partitions now renamed into last_data_modified_time for clarity. Warnings might be produced with old attribute names.

Features

• Add new command line tool pyodps-pack to pack third-party libraries, recommended as standard packing mechanism
• (Experimental) Add preliminary support for custom DataFrame functions with Python 3.8 / 3.9 / 3.10
• Supports DataFrame column join methods
• Support configuring instance settings via connection strings with SQLAlchemy
• (Experimental) Supports external volume
• Supports run_sql_interactive_with_fallback interface in pyodps
• Supports get_max_partition for tables
• Implements stream read and write of resources

Enhancements

• Supports quota name when creating tunnels
• Allow apply UDFs to use __getitem__
• Reset table object when alter table is called
• Allow persisting DataFrames with table objects
• Support compression with LZ4 and zstd for tunnels

Bug fixes

• Fix downloading with multiprocessing under Windows
• Fix negative timestamp issue under Windows
• Fix multiple issues on Github
• Resolve coded json when parsing
• Add retry when encountering MetaTransactionFailed
• Fix parsing RFC822 date when loading ODPS meta
• Fix schema error when calling to_pandas() with columns specified

Compatibility Issues

• As new Schema object level is introduced, it is now discouraged to use schema for table schemas and warnings will be produced. Try using table_schema instead when you code with the new version.
• Attributes like creation_time now uses local time instead of UTC time for consistency with other datetime attributes by default. Switch to old behavior by setting options.use_legacy_parsedate = True.

Bugfixes

1. Add retry on tunnel meta conflicts.
2. Fix forward compatibility for v0.11.3 on table schema arguments.

• Supports interactive query with retry
• Bug fixes

• Supports security queries returning instances
• Bug fixes

• Make public API for arrow tunnels
• Add support for skew join
• Bug & docs fixes

• Upgrade Mars support to v0.9.0 and switch to different bases for different Python versions and archs
• Accelerate tunnel read (Thanks @torshie )
• Fix tests & docs.

• Add support for Mars 0.8.0
• Fix compatibility for IPython>=0.8.0
• Use cibuildwheel to publish aarch64 wheels as well as Python 3.10 wheels.

• Add sts account support.
• Fix to_pandas error on Windows.

• Add more sqa features to support superset
• Add docs for downloading with multiprocessing

• Use split meta to estimate chunk size
• Support specifying a predicate when reading partitions
• Allow suspending ipywidgets by options
• Fix persist to existed partition

• Apply head optimization for read table
• Support reading whole partitioned table
• Implement arrow tunnel

• Config UDF python version if has UDF in query
• add biz_id meta to xflow instance
• Fix errors when import tensorflow in Mars
• Some enhancements

• doc enhancements
• Fix reduce kwargs in pandas backend

• Fixes for SQLAlchemy.
• Add volume filesystem support.
• Switch deployment to Github Actions.
• Add reserved slot in partition schema.
• Optimized error handling logic and interface for session.

Enhancements

• Add support for StsAccount.

• Upgrade Mars to 0.5.2
• Support rescale workers
• Fix memory issues

• Add sqlalchemy support
• Add refresh mechanism for bearer token
• Fix behavior of persist Mars dataframe

• Improvements and bug fixes for Mars integration
• Add stored_as property for Table
• Fix some compatibility issues
• Documentation improvements

• Add archive table support.
• Enhancements in Mars integration.

• Enhancements in Mars integration.
• Documentation enhancement.
• Bugfixes.

Enhancements

• Add support for StsAccount.

• Fix pickle protocol inconsistent
• Add list_projects support
• Bugfixes

• Support running DataFrame customized functions under Python 3 backends
• Support date type
• Get logview host address automatically
• Support async create download session
• Add model copy support
• bugfix

• Support pandas 0.25+.
• Support execute_merge_files to merge small files.