pynamodb

Fixes:

Fix a warning about datetime.utcfromtimestamp deprecation ( #1261 )

This is a major release and contains breaking changes. Please read the notes below carefully.

Breaking changes:

BinaryAttribute and BinarySetAttribute have undergone breaking changes:

The attributes’ internal encoding has changed. To prevent this change going unnoticed, a new required legacy_encoding parameter was added: see Upgrading Binary(Set)Attribute for details. If your codebase uses BinaryAttribute or BinarySetAttribute , go over the attribute declarations and mark them accordingly.

When using binary attributes, the return value of serialize() will no longer be JSON-serializable since it will contain bytes objects. Use to_dynamodb_dict() and to_simple_dict() for JSON-serializable mappings. for a safe JSON-serializable representation.

Python 3.6 is no longer supported.

PynamoDB no longer has a default AWS region (used to be us-east-1) ( #1003 ). If needed, update your models’ Meta or set the AWS_DEFAULT_REGION environment variable.

Model ’s JSON serialization helpers were changed:

to_json was renamed to to_simple_dict() ( #1126 ). Additionally, to_dynamodb_dict() and from_dynamodb_dict() were added for round-trip JSON serialization.

pynamodb.util.attribute_value_to_json was removed ( #1126 )

Attribute ’s default parameter must be either an immutable value (of one of the built-in immutable types) or a callable. This prevents a common class of errors caused by unintentionally mutating the default value. A simple workaround is to pass an initializer (e.g. change default={} to default=dict ) or wrap in a lambda (e.g. change default={'foo': 'bar'} to default=lambda: {'foo': 'bar'} ).

count() , query() , and scan() are now instance methods.

OperationSettings has been removed.

Major changes:

We are now compatible with opentelemetry botocore instrumentation .

We’ve reduced our usage of botocore private APIs ( #1079 ). On multiple occasions, new versions of botocore broke PynamoDB, and this change lessens the likelihood of that happening in the future by reducing (albeit not eliminating) our reliance on private botocore APIs.

Minor changes:

save() , update() , delete_item() , and delete() now accept a add_version_condition parameter. See Conditioning on the version for more details.

batch_get() , has guard rails defending against items without a hash_key and range_key.

set() , can remove attribute by assigning an empty value in the update expression.

This is a major release and contains breaking changes. Please read the notes below carefully.

Breaking changes:

• ~pynamodb.attributes.BinaryAttribute and ~pynamodb.attributes.BinarySetAttribute have undergone breaking changes:

  • The attributes' internal encoding has changed. To prevent this change going unnoticed, a new required legacy_encoding parameter was added: see upgrading_binary for details. If your codebase uses ~pynamodb.attributes.BinaryAttribute or ~pynamodb.attributes.BinarySetAttribute, go over the attribute declarations and mark them accordingly.

  • When using binary attributes, the return value of ~pynamodb.models.Model.serialize will no longer be JSON-serializable since it will contain bytes objects. Use ~pynamodb.attributes.AttributeContainer.to_dynamodb_dict and ~pynamodb.attributes.AttributeContainer.to_simple_dict for JSON-serializable mappings. for a safe JSON-serializable representation.

• Python 3.6 is no longer supported.

• PynamoDB no longer has a default AWS region (used to be us-east-1) (1003). If needed, update your models' Meta or set the AWS_DEFAULT_REGION environment variable.

• ~pynamodb.models.Model's JSON serialization helpers were changed:

  • to_json was renamed to ~pynamodb.attributes.AttributeContainer.to_simple_dict (1126). Additionally, ~pynamodb.attributes.AttributeContainer.to_dynamodb_dict and ~pynamodb.attributes.AttributeContainer.from_dynamodb_dict were added for round-trip JSON serialization.

  • pynamodb.util.attribute_value_to_json was removed (1126)

• ~pynamodb.attributes.Attribute's default parameter must be either an immutable value (of one of the built-in immutable types) or a callable. This prevents a common class of errors caused by unintentionally mutating the default value. A simple workaround is to pass an initializer (e.g. change default={} to default=dict) or wrap in a lambda (e.g. change default={'foo': 'bar'} to default=lambda: {'foo': 'bar'}).

• ~pynamodb.indexes.Index.count, ~pynamodb.indexes.Index.query, and ~pynamodb.indexes.Index.scan are now instance methods.

• ~pynamodb.settings.OperationSettings has been removed.

Major changes:

• We are now compatible with opentelemetry botocore instrumentation.

• We've reduced our usage of botocore private APIs (1079). On multiple occasions, new versions of botocore broke PynamoDB, and this change lessens the likelihood of that happening in the future by reducing (albeit not eliminating) our reliance on private botocore APIs.

Minor changes:

• ~pynamodb.models.Model.save, ~pynamodb.models.Model.update, ~pynamodb.models.Model.delete_item, and ~pynamodb.models.Model.delete now accept a add_version_condition parameter. See optimistic_locking_version_condition for more details.

• ~pynamodb.models.Model.batch_get, has guard rails defending against items without a hash_key and range_key.

• ~pynamodb.attributes.Attribute.set, can remove attribute by assigning an empty value in the update expression.

Fix compatibility with botocore 1.33.2 (#1205)

save() , update() , delete_item() , and delete() now accept a add_version_condition parameter. See Conditioning on the version for more details.

• ~pynamodb.models.Model.save, ~pynamodb.models.Model.update, ~pynamodb.models.Model.delete_item, and ~pynamodb.models.Model.delete now accept a add_version_condition parameter. See optimistic_locking_version_condition for more details.

Use model’s AWS credentials in threads (#1164)

A model can specify custom AWS credentials in the Meta class (in lieu of “global” AWS credentials from the environment). Previously those model-specific credentials were not used from within new threads.

Contributors to this release:

@atsuoishimoto

Expose transaction cancellation reasons in cancellation_reasons() and cancellation_reasons() (#1144).

• Expose transaction cancellation reasons in ~pynamodb.exceptions.TransactWriteError.cancellation_reasons and ~pynamodb.exceptions.TransactGetError.cancellation_reasons (#1144).

Prevent typing_tests from being installed into site-packages ( #1118 )

Contributors to this release:

@musicinmybrain

Fixed issue introduced in 5.3.0: using TableConnection directly (not through a model) raised the following exception:

pynamodb . exceptions . TableError : Meta - table for '(table-name)' not initialized

Fix typing on TransactGet (backport of #1057 )

• Fixed issue introduced in 5.3.0: using ~pynamodb.connection.table.TableConnection directly (not through a model) raised the following exception:

  pynamodb.exceptions.TableError: Meta-table for '(table-name)' not initialized

• Fix typing on ~pynamodb.transactions.TransactGet (backport of 1057)

No longer call DescribeTable API before first operation

Before this change, we would call DescribeTable before the first operation on a given table in order to discover its schema. This slowed down bootstrap (particularly important for lambdas), complicated testing and could potentially cause inconsistent behavior since queries were serialized using the table’s (key) schema but deserialized using the model’s schema.

With this change, both queries and models now use the model’s schema.

Update for botocore 1.28 private API change ( #1083 ) which caused the following exception:

TypeError : _convert_to_request_dict () missing 1 required positional argument : 'endpoint_url'

Fix issue from 5.2.0 with attempting to set GSI provisioned throughput on PAY_PER_REQUEST billing mode ( #1018 )

The IndexMeta class has been removed. Now type(Index) == type ( #998 )

JSON serialization support ( Model.to_json and Model.from_json ) has been added ( #857 )

Improved type annotations for expressions and transactions ( #951 , #991 )

Always use Model attribute definitions in create table schema ( #996 )

date 2021-02-11

Do not serialize all attributes for updates and deletes ( #905 )

date 2021-01-26

This is major release and contains breaking changes. Please read the notes below carefully.

Breaking changes:

Python 2 is no longer supported. Python 3.6 or greater is now required.

UnicodeAttribute and BinaryAttribute now support empty values ( #830 )

In previous versions, assigning an empty value to would be akin to assigning None : if the attribute was defined with null=True then it would be omitted, otherwise an error would be raised.

As of May 2020, DynamoDB supports empty values for String and Binary attributes. This release of PynamoDB starts treating empty values like any other values. If existing code unintentionally assigns empty values to StringAttribute or BinaryAttribute, this may be a breaking change: for example, the code may rely on the fact that in previous versions empty strings would be “read back” as None values when reloaded from the database.

UTCDateTimeAttribute now strictly requires the date string format '%Y-%m-%dT%H:%M:%S.%f%z' to ensure proper ordering. PynamoDB has always written values with this format but previously would accept reading other formats. Items written using other formats must be rewritten before upgrading.

Table backup functionality ( Model.dump[s] and Model.load[s] ) has been removed.

Model.query no longer converts unsupported range key conditions into filter conditions.

Internal attribute type constants are replaced with their “short” DynamoDB version ( #827 )

Remove ListAttribute.remove_indexes (added in v4.3.2) and document usage of remove for list elements ( #838 )

Remove pynamodb.connection.util.pythonic ( #753 ) and ( #865 )

Remove ModelContextManager class ( #861 )

Features:

Polymorphism

This release introduces Polymorphism support via DiscriminatorAttribute . Discriminator values are written to DynamoDB and used during deserialization to instantiate the desired class.

Model Serialization

The Model class now includes public methods for serializing and deserializing its attributes. Model.serialize and Model.deserialize convert the model to/from a dictionary of DynamoDB attribute values.

Other changes in this release:

Typed list attributes can now support any Attribute subclass ( #833 )

Most API operation methods now accept a settings argument to customize settings of individual operations. This currently allow adding or overriding HTTP headers. ( #887 )

Add the attribute name to error messages when deserialization fails ( #815 )

Add the table name to error messages for transactional operations ( #835 )

Contributors to this release:

@jpinner

@ikonst

@rchilaka -amzn

@jonathantan

Fix compatibility with botocore 1.33.2 (#1235)

Update for botocore 1.28 private API change (#1130) which caused the following exception:

TypeError : _convert_to_request_dict () missing 1 required positional argument : 'endpoint_url'

Add type stubs for indexing into a ListAttribute for forming conditional expressions ( #774 )

class MyModel ( Model ): ... my_list = ListAttribute () MyModel . query ( ... , condition = MyModel . my_list [ 0 ] == 42 )

Fix a warning about collections.abc deprecation ( #782 )

Fix discrepancy between runtime and type-checker’s perspective of Index and derived types ( #769 )

Add ListAttribute.remove_indexes action for removing specific indexes from a ListAttribute ( #754 )

Type stub fixes:

Add missing parameters of Model.scan ( #750 )

Change Model.get ’s hash_key parameter to be typed Any ( #756 )

Prevent integration tests from being packaged ( #758 )

Various documentation fixes ( #762 , #765 , #766 )

Contributors to this release:

@mxr

@sodre

@biniow

@MartinAltmayer

@dotpmrcunha

@meawoppl

Fix Index.query and Index.scan typing regressions introduced in 4.2.0, which were causing false errors in type checkers

Implement exponential backoff for batch writes ( #728 )

Avoid passing ‘PROVISIONED’ BillingMode for compatibility with some AWS AZs ( #721 )

On Python >= 3.3, use importlib instead of deprecated imp ( #723 )

Update in-memory object correctly on REMOVE update expressions ( #741 )

Contributors to this release:

@hallie

@bit -bot-bit

@edholland

@reginalin

@MichelML

@timgates42

@sunaoka

@conjmurph

• Implement exponential backoff for batch writes (728)

• Avoid passing 'PROVISIONED' BillingMode for compatibility with some AWS AZs (721)

• On Python >= 3.3, use importlib instead of deprecated imp (723)

• Update in-memory object correctly on REMOVE update expressions (741)

Contributors to this release:

• hallie

• bit-bot-bit

• edholland

• reginalin

• MichelML

• timgates42

• sunaoka

• conjmurph

date 2019-10-17

This is a backwards compatible, minor release.

Add attributes_to_get parameter to Model.scan ( #431 )

Disable botocore parameter validation for performance ( #711 )

Contributors to this release:

@ButtaKnife

date 2019-04-10

This is major release and contains breaking changes. Please read the notes below carefully.

Requests Removal

Given that botocore has moved to using urllib3 directly for making HTTP requests, we’ll be doing the same (via botocore ). This means the following:

The session_cls option is no longer supported.

The request_timeout_seconds parameter is no longer supported. connect_timeout_seconds and read_timeout_seconds are available instead.

Note that the timeouts for connection and read are now 15 and 30 seconds respectively. This represents a change from the previous 60 second combined requests timeout.

Wrapped exceptions (i.e exc.cause ) that were from requests.exceptions will now be comparable ones from botocore.exceptions instead.

Key attribute types must match table

The previous release would call DescribeTable to discover table metadata and would use the key types as defined in the DynamoDB table. This could obscure type mismatches e.g. where a table’s hash key is a number ( N ) in DynamoDB, but defined in PynamoDB as a UnicodeAttribute .

With this release, we’re always using the PynamoDB model’s definition of all attributes including the key attributes.

Deprecation of old APIs

Support for Legacy Conditional Parameters has been removed. See a complete list of affected Model methods below:

Method

Changes

update_item

removed in favor of update

rate_limited_scan

removed in favor of scan and ResultIterator

delete

conditional_operator and **expected_values kwargs removed. Use condition instead.

update

attributes , conditional_operator and **expected_values kwargs removed. Use actions and condition instead.

save

conditional_operator and **expected_values kwargs removed. Use condition instead.

count

**filters kwargs removed. Use range_key_condition / filter_condition instead.

query

conditional_operator and **filters kwargs removed. Use range_key_condition / filter_condition instead.

scan

conditional_operator and **filters kwargs removed. Use filter_condition instead.

allow_rate_limited_scan_without_consumed_capacity was removed

When upgrading, pay special attention to use of **filters and **expected_values , as you’ll need to check for arbitrary names that correspond to attribute names. Also keep an eye out for kwargs like user_id__eq=5 or email__null=True , which are no longer supported. A type check can help you catch cases like these.

New features in this release:

Support for transactions ( TransactGet and TransactWrite ) ( #618 )

Support for versioned optimistic locking ( #664 )

Other changes in this release:

Python 2.6 is no longer supported. 4.x.x will be the last major release to support Python 2.7 given the upcoming EOL.

Added the max_pool_connection and extra_headers settings to replace common use cases for session_cls

Added support for moto through implementing the botocore “before-send” hook.

Performance improvements to UTCDateTimeAttribute deserialization. ( #610 )

The MapAttributeMeta class has been removed. Now type(MapAttribute) == AttributeContainerMeta .

Removed LegacyBooleanAttribute and the read-compatibility for it in BooleanAttribute .

None can now be used to bootstrap condition chaining ( #653 )

Allow specifying timedeltas in expressions involving TTLAttributes ( #665 )

date 2019-06-13

This is a backwards compatible, minor release.

Changes in this release:

Adds a TTLAttribute that specifies when items expire ( #259 )

Enables time-to-live on a DynamoDB table if the corresponding model has a TTLAttribute

Adds a default_for_new parameter for Attribute which is a default that applies to new items only

Contributors to this release:

@irhkang

@ikonst

date 2019-01-15

This is a backwards compatible, minor release.

Fixes in this release:

Legacy boolean attribute migration fix. ( #538 )

Correctly package type stubs. ( #585 )

Contributors to this release:

@vo -va

date 2019-01-03

This is a backwards compatible, minor release.

Changes in this release:

Built-in support for mypy type stubs, superseding those in python/typeshed. ( #537 )

date 2018-08-30

This is a backwards compatible, minor bug fix release.

Fixes in this release:

Clearer error message on missing consumed capacity during rate-limited scan. ( #506 )

Python 3 compatibility in PageIterator. ( #535 )

Proxy configuration changes in botocore>=1.11.0. ( #531 )

Contributors to this release:

@ikonst

@zetaben

@ningirsu

date 2018-05-09

This is a backwards compatible, major bug fix release.

New features in this release:

Support scan operations on secondary indexes. ( #141 , #392 )

Support projections in model get function. ( #337 , #403 )

Handle values from keys when batch get returns unprocessed keys. ( #252 , #376 )

Externalizes AWS Credentials. ( #426 )

Add migration support for LegacyBooleanAttribute. ( #404 , #405 )

Rate limited Page Iterator. ( #481 )

Fixes in this release:

Thread-safe client creation in botocore. ( #153 , #393 )

Use attr.get_value(value) when deserialize. ( #450 )

Skip null attributes post serialization for maps. ( #455 )

Fix deserialization bug in BinaryAttribute and BinarySetAttribute. ( #459 , #480 )

Allow MapAttribute instances to be used as the RHS in expressions. ( #488 )

Return the correct last_evaluated_key for limited queries/scans. ( #406 , #410 )

Fix exclusive_start_key getting lost in PageIterator. ( #421 )

Add python 3.5 for Travis ci builds. ( #437 )

Contributors to this release:

@jpinner -lyft

@scode

@behos

@jmphilli

@drewisme

@nicysneiros

@jcomo

@kevgliss

@asottile

@harleyk

@betamoo

date 2017-10-25

This is a backwards compatible, minor bug fix release.

Removed features in this release:

Remove experimental Throttle api. ( #378 )

Fixes in this release:

Handle attributes that cannot be retrieved by getattr. Fixes #104 ( #385 )

Model.refresh() should reset all model attribuets. Fixes #166 ( #388 )

Model.loads() should deserialize using custom attribute names. Fixes #168 ( #387 )

Deserialize hash key during table loads. Fixes #143 ( #386 )

Support pagination in high-level api query and scan methods. Fixes #50 , #118 , #207 , and #248 ( #379 )

Don’t serialize null nested attributed. Fixes #240 and #309 ( #375 )

Legacy update item subset removal using DELETE operator. Fixes #132 ( #374 )

Contributors to this release:

@jpinner -lyft

date 2017-10-13

This is a backwards compatible, minor release.

This release updates PynamoDB to interact with Dynamo via the current version of Dynamo’s API. Condition and update expressions can now be created from attributes and used in model operations. Legacy filter and attribute update keyword arguments have been deprecated. Using these arguments will cause a warning to be logged.

New features in this release:

Add support for current version of DynamoDB API

Improved MapAttribute item assignment and access.

Contributors to this release:

@jpinner -lyft

date 2017-10-25

This is a backwards compatible, minor release.

The purpose of this release is to prepare users to upgrade to v3.0.1+ (see issue #377 for details).

Pull request #294 removes the backwards compatible deserialization of UnicodeSetAttributes introduced in #151 .

This release introduces a migration function on the Model class to help re-serialize any data that was written with v1.5.4 and below.

Temporary feature in this release:

Model.fix_unicode_set_attributes() migration helper

Model.needs_unicode_set_fix() migration helper