mypy-protobuf

• prep 3.8.0 release

Signed-off-by: Aidan Jensen aidandj.github@gmail.com

• bump to 4.0.0

Signed-off-by: Aidan Jensen aidandj.github@gmail.com

Signed-off-by: Aidan Jensen aidandj.github@gmail.com

• Mark top-level mangled identifiers as TypeAlias.
• Change the top-level mangling prefix from global___ to Global___ to respect Y042 naming convention.
• Support client stub async typing overloads
• Support PEP702 deprecations
  • Message deprecations are supported
  • Field deprecations are not. This may be possible with init overloads
  • Service deprecations are supported for Sync stubs
    • Not for async stubs
  • Enum message deprecation is supported
    • Enum field deprecation is not
• Drop Python 3.8 testing. Newer protobuf versions are incompatible. Generated code may still work

Bump to 3.6.0 (#584)

Bump to 3.5.0 (#541)

Bump to 3.4.0 (#440)

Bump to release 3.3.0 (#401)

Bump to release 3.2.0 (#343)

Bump to release 3.1.0 (#323)

• Drop support for targeting python 2.7
• Generate py3 specific syntax for unicode strings ("" rather than u"")
• Now requires protobuf 3.18.0
• Handle escaping properly in docstrings and attribute strings (#296)
• Use three-digit version number 3.0.0 for minor and patch releases
• Codify pyright support in README

• Switch from setup.py to pyproject.toml and setup.cfg per https://packaging.python.org/tutorials/packaging-projects/
• Remove dependency on grpcio-tools. mypy-protobuf doesn't need it to run. It's only needed to run mypy afterward.
• Avoid relative imports in _grpc_pb2.pyi stubs. grpc stubs themselves don't use relative imports, nor __init__.py files
• Use """ docstring style comments in generated code rather than # style so it shows up in IDEs.
• Disambiguate messages from reserved python keywords (eg None) with prefix _r_ rather than __ - since __ is considered private.
• Bump tests to use pyright 1.1.169, types-protobuf 3.17.4, pytest 6.2.5, grpc-stubs 1.24.7, grpcio-tools 1.40.0
• Since upstream protobuf has dropped support with 3.18.0, 2.10 will be the last mypy-protobuf that supports targeting python 2.7. Updated docs for this.

• Rename master branch to main
• Make install_requires requirements for grpcio-tools and types-protobuf use >=

• Propagate comments from .proto files to .pyi files
• Add protobuf type stubs to the setup requirements
• Fix #239 Remove type: ignore used in enum by pulling V into a separate class.
• Use pytest 6.2.4 for internal test suites on python3
• Remove protoc_gen_mypy.bat as the entry-points method creates protoc-gen-mypy.exe. Add test confirming.

• Fix #244 - support extensions defined at module scope with proper types, matching extensions defined within Messages. See _ExtensionDict

extend google.protobuf.MessageOptions {
   string test_message_option = 51234;
}

# Used to generate
test_message_option: google.protobuf.descriptor.FieldDescriptor = ...

# Now generates
test_message_option: google.protobuf.internal.extension_dict._ExtensionFieldDescriptor[google.protobuf.descriptor_pb2.MessageOptions, typing.Text] = ...

Fix repeated extensions as well - to generate RepeatedScalarFieldContainer and RepeatedCompositeFieldContainer

• Now requires types-protobuf 3.17.3
• Fix #238 - handling enum variants that name conflict with EnumTypeWrapper methods
• Improve mypy-protobuf testsuite expected-errors file to make insertions/deletions easier to code review
• Fix #227 - improve support for messages/enums with python reserved keyword names
• Order fields within a message in original .proto file order Previously, they were grouped by scalar/nonscalar. Remove that grouping to make it a bit easier to correlate .proto files to .pyi files.

• Bump protoc support to 3.17.3
• Use latest python versions in tests (3.6.14 3.7.11 3.8.11 3.9.6)
• Support reserved names for message types. Previously generated invalid mypy.

message M {
  message None {}
  None none = 1;
}

• Support protoc-gen-mypy -V and protoc-gen-mypy --version to print version number
• Return Optional[Literal[...]] instead of Literal[...] from WhichOneof to support cases in which none of the fields of the WhichOneof are set. See the following example.

def hello(name: str) -> None: ...
n = proto.WhichOneof("name")
hello(n)  # Will now result in a mypy error.
assert n is not None
hello(n)  # Should work ok

• Bump mypy version to 0.910, utilizing stubs types-protobuf==0.1.14. See https://mypy-lang.blogspot.com/2021/05/the-upcoming-switch-to-modular-typeshed.html
• Bump grpcio version tested to 1.38.1 and grpc-stubs to 1.24.6
• Generate a # type: ignore for enum generated stubs to avoid circular dependency described in #214. Bandaid solution.

• Organized generated enum code to prevent definition ordering issues in Pyright-based linters
• Changed type generation for grpcio stubs to use the MultiCallable API (see here) . This requires using the grpc-stubs typings for grpcio. This change should allow calling stub methods with common parameters (timeout, metadata, etc.) as well as calling methods on the MultiCallable object (e.g. my_stub.MyRpcMethod.future()).
• Update stubs to mark repeated scalar and repeated enum fields as read-only

• Add support for _FIELD_NUMBER generated fields on messages as specified in the spec

• Fix CI caching across version updates.
• Added unit testing for EnumTypeWrapper.Value usage
• Reexport of top level enum values when using import public

• Fix bug where module level DESCRIPTOR: FileDescriptor was not being generated for files w public imports

• Fix crash when a import public reexport when dependent proto file is not listed on command line for generation

Non Backward Compatible Changes

• Dropping support for running mypy-protobuf in python <= 3.5. Note you can still generate stubs target-compatible to python2
• Type proto Enum values for as MyEnum.V rather than MyEnumValue for import ergonomics, allowing the caller to import MyEnum rather than conditionally importing MyEnumValue
• Default disallow None as argument for primitive fields of constructors in proto3. Provided relax_strict_optional_primitives flag to relax this strictness if you prefer.

New Features

• Support for grpcio stubs generation
• Allow mypy_protobuf.py to be run directly as a script
• Add support for proto's well_known_types
• Support message fields named self - by renaming the constructor's self to self_
• Rename extensions proto from mypy/mypy.proto to mypy_protobuf/extensions.proto
• Add support for mypy-proto extensions mypy_protobuf.casttype, mypy_protobuf.keytype, and mypy_protobuf.valuetype
• Add support for import public proto imports - by reexporting in generated code

Output Format

• Generate fully qualified references rather than mangling
• Import builtins library rather than mangling builtins
• Use fully qualified names rather than mangling imports
• Only mangle-alias top-level identifiers w/ global___ to avoid conflict w/ fields of same name [previously was mangling inner messages as well]
• Add support for readable_stubs parameter for mangle-free output code w/o fully-qualified references to message. (in many cases this is good enough and easier to read)
• Generate arg: Optional[type] = ... instead of arg: Optional[type] = None
• Avoid importing google.protobuf.message.Message unless it's needed

Internal Improvements

• Add support for python 3.9 to CI
• Update mypy-protobuf CI to target 3.8 rather than 3.5
• Inline mypy annotations, eliminate six, and remove __future__ import in mypy_protobuf_lib.py
• Flatten directory structure (remove python subdirectory). Updated unreleased installation instructions.
• Deprecate and delete the go/ implementation

• Bump required mypy version to 0.800
• Bump required protobuf version to 3.14 (pi!!)
• Update the autogenerated .pyi file header to cite mypy-protobuf
• Reorganize mypy-protobuf testsuite files to more closely match autogeneration into a generated directory
• Remove incorrect type___ prefixed alias for inner enum type wrappers
• Add support for extension fields in proto2 messages
• Overwrite typing for Message.Extensions mapping to support better type inference for proto2 message extensions
• Support Message.HasExtension and Message.ClearExtension
• Bump python-protobuf from 3.11.3 to 3.13.0
• Add support for optional proto3 fields
• Support ScalarMap and MessageMap generated types for map types in proto. This will allow us to support get_or_create

message Message {
    map<int32, OuterMessage3> map_message = 17
}

and

message.map_message.get_or_create(0)

Before (1.23)

main.py:4: error: "MutableMapping[str, Nested]" has no attribute "get_or_create"  [attr-defined]

After (1.24) - there is no error

• Inherit FromString from superclass Message - rather than re-generating here. Fixes bug in python2 usage google/protobuf/type_pb2.pyi:92: error: Argument 1 of "FromString" is incompatible with supertype "Message"; supertype defines the argument type as "ByteString" [override]

• Update tested/required mypy version to 0.780 (picks up new typeshed annotations). Includes improved typing/error messages on Message.

Before (mypy < 0.780):

test_negative/negative.py:26: error: Argument 1 to "CopyFrom" of "Message" has incompatible type "str"; expected "Message"

After (mypy >= 0.780:

test_negative/negative.py:26: error: Argument 1 to "CopyFrom" of "Message" has incompatible type "str"; expected "Simple1"

• Update generated EnumTypeWrapper to be instances of EnumTypeWrapper - for more consistency with generated python code. Most caller code should not require mypy type changes. Egh ProtoEnum.Value('first') should work either way.

Generated Before (in 1.21)

class ProtoEnum(object):
    @classmethod
    def Value(cls, name: str) -> ProtoEnumValue

Generated After (in 1.22)

ProtoEnum: _ProtoEnum
class _ProtoEnum(google.protobuf.EnumTypeWrapper):
    def Value(self, name: str) -> ProtoEnumValue

• Remove autogenerated EnumTypeWrapper methods that are redundant to the typeshed parent class. Added testing for these.

• Support for module descriptor.
• Update mangling from global__ to message__
• Fix bug in message typing for nested enums. Split EnumValue from EnumTypeWrapper. Enforces that constructing an enum value must happen via a NewType wrapper to the int.

Example:

enum ProtoEnum {
    FIRST = 1;
    SECOND = 2;
}

mesage ProtoMsg {
    ProtoEnum enum = 1;
}

Generated Before (in 1.20):

class ProtoEnum(object):
    @classmethod
    def Value(cls, name: str) -> ProtoEnum

class ProtoMsg(Message):
    def __init__(self, enum: ProtoEnum) -> None

Generated After (in 1.21):

ProtoEnumValue = NewType('ProtoEnumValue', int)
class ProtoEnum(object):
    @classmethod
    def Value(cls, name: str) -> ProtoEnumValue

class ProtoMsg(Message):
    def __init__(self, enum: ProtoEnumValue) -> None

Migration Guide (with example calling code)

Before (with 1.20)

from msg_pb2 import ProtoEnum, ProtoMsg

def make_proto_msg(enum: ProtoEnum) -> ProtoMsg:
    return ProtoMsg(enum)
make_proto_msg(ProtoMsg.FIRST)

After (with 1.21)

from msg_pb2 import ProtoEnum, ProtoMsg

def make_proto_msg(enum: 'msg_pb2.ProtoEnumValue') -> ProtoMsg:
    return ProtoMsg(enum)
make_proto_msg(ProtoMsg.FIRST)

• Use inline-style rather than comment-style typing in the pyi file
• Remove MergeFrom/CopyFrom from generated code as it is in the Message superclass

• Black code formatting
• Fix message/field name aliasing when field name matches a message/enum name

• Allow omitting required proto2 fields from constructor parameters
• Support and testing for python 3.8
• Support for python-protobuf to 3.11.3

• Use entry_points:console_scripts to support long paths to the python interpreter

• Update to newer mypy version - including minor changes to typeshed

• Absolute path to necessary python
• Add forward reference string literal support for enums
• Alias builtin types to avoid collision with field name

• Add class to set of python keywords

• Add Message.DESCRIPTOR