Getting Started¶
Installation¶
required: python >= 3.7
install from PyPi via
pip install codestare-msg-compileror check out source and install locally.Set the plugin as a build dependency in your
pyproject.tomle.g.
[build-system]
requires = [
"setuptools>=42",
"wheel",
"codestare-msg-compiler[protoplus] >= 0.0.4",
]
...
There are extras defined for the different supported flavors
of python protobuf messages, to easily install all required dependencies (e.g. third party protoc plugins).
The available extras for python-package flavors are:
[mypy](depends on / installsmypy,mypy-protobuf)[betterproto](depends on / installsbetterproto[compiler])[protoplus](depends on / installscodestare-proto-plus)
CLI Usage¶
If installed with extra [cli] the package provides two command line entry points:
Available options for compilation / rewriting files and respective arguments can be found using the --help flag of the
cli tools and in the API documentation
For example:
$ compile-proto --help
INFO: Showing help with the command 'compile-proto -- --help'.
NAME
compile-proto - Wrapper around protobuf compiler.
SYNOPSIS
compile-proto GROUP | <flags>
DESCRIPTION
See ``compile-proto OPTIONS`` for all possible compilation options.
See ``compile-proto`` (no args) and ``compile-proto --help`` for more help
FLAGS
--protoc=PROTOC
Default: '/usr/bin/protoc'
GROUPS
GROUP is one of the following:
OPTIONS
You can then show the available options:
$ compile-proto OPTIONS
[java] java
[cs] csharp
[cpp] cpp
[better] python with `better_proto` plugin
[mypy] mypy python stubs
[plus] python_protoplus
[py] python
[js] javascript as library | javascript individual
[all] java | javascript as library | javascript individual | csharp | cpp | python with `better_proto` plugin | mypy python stubs | python_protoplus | python
Setup¶
The package installs setuptools commands.:
$ python setup.py --help-commands
...
compile_python compile python protobuf modules (google plugin)
generate_inits generate (better) __init__.py files for protobuf modules
rewrite_proto rewrite protobuf inputs to fake specific package structure (experimental)
compile_catalog compile message catalogs to binary MO files
extract_messages extract localizable strings from the project code
init_catalog create a new catalog based on a POT file
update_catalog update message catalogs from a POT file
build_sphinx Build Sphinx documentation
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
To build your project with the extra build steps, use a custom
setup.py file, e.g.
import sys
import setuptools
try:
from importlib import metadata
except ImportError: # for Python<3.8
import importlib_metadata as metadata
build_proto_name = 'build_py_proto'
build_cmds = [c for c in metadata.entry_points()['distutils.commands']
if c.name == build_proto_name]
if not build_cmds:
print(f"No distutils command with name {build_proto_name} found."
f" Did you install the `ubii-msg-compiler` package?")
sys.exit(1)
else:
build_cmds = list(set(build_cmds))
assert len(build_cmds) == 1
build_py = build_cmds[0].load()
setuptools.setup(
cmdclass={
'build_py': build_py,
build_proto_name: build_py,
}
)
Note
The example project is available for download.
You might want to test the behaviour of the build commands on some files that
you don’t care about ;)
You can then configure the build steps via your setup.cfg file.
Let’s say your project has the following structure:
$ tree ./example_project
./example_project
├── build
├── pyproject.toml
├── setup.cfg
├── setup.py
└── src
├── proto
│ ├── general
│ │ ├── error.proto
│ │ └── success.proto
│ └── services
│ ├── reply.proto
│ ├── request
│ │ └── subscription.proto
│ ├── request.proto
│ └── service.proto
└── py
└── namespace
└── proto
├── __init__.py
├── additonal_module.py
└── v1
10 directories, 11 files
Note
To make things more interesting the .proto files are not included in the
python package yet (notice the package_dir option in the setup.cfg).
Maybe they are shared between different projects, some of which
are not even written in python – src/proto could just be a git submodule
Also, the protobuf package should be built to src/py/namespace/proto/v1.
You want to import all message types from there in the __init.py__ of your namespace.proto package,
to create a simpler (“flat”) API for your users and to make it easy to manage several
versions of the protobuf schema – by simply importing from a different subpackage, if needed.
To achieve this, configure your build steps like this (this setup uses the proto-plus flavor,
so you need to install the [protoplus] extra):
[metadata]
name = example-project
author = ...
author_email = foo@bar.com
description = "Documentation Example"
classifiers =
Programming Language :: Python :: 3
[options]
python_requires = >=3.7
setup_requires =
codestare-msg-compiler >= 0.0.4a1dev5
importlib_metadata;python_version<"3.8"
install_requires =
proto-plus >= 1.19.0
package_dir =
= src/py
packages = find_namespace:
[options.packages.find]
where = src/py
# define include directory with .proto files
[build_py_proto]
include_proto = src/proto
# set build directory and enforce python package
# choose your flavor
[compile_proto]
build_lib = src/py
proto_package = namespace.proto.v1
flavor = plus
Now build your project…:
$ cd example_project/
$ python setup.py build
running build
running build_py
running compile_proto
Including *.proto files from path[s] [PosixPath('src/proto')]
running rewrite_proto
Enforcing python package namespace.proto.v1 for compiled modules.
...
What has been generated?
$ tree ./example_project/src/py
./example_project/src/py
└── namespace
└── proto
├── __init__.py
├── additonal_module.py
└── v1
├── general
│ ├── error.proto
│ ├── error_pb_plus.py
│ ├── success.proto
│ └── success_pb_plus.py
└── services
├── reply.proto
├── reply_pb_plus.py
├── request
│ ├── subscription.proto
│ └── subscription_pb_plus.py
├── request.proto
├── request_pb_plus.py
├── service.proto
└── service_pb_plus.py
6 directories, 14 files
Warning
The compiler tools have copied the “fixed” version of the .proto files to
the python package, so you can inspect what was changed. When you set the
inplace option (e.g. in your
setup.cfg) the proto files will not be copied to the python package,
and instead be overwritten. Only use this feature if your .proto sources
are under version control and you can correct possible errors easily!
For example the original .proto file looked like this …
$ cat ./example_project/src/proto/general/error.proto
syntax = "proto3";
package namespace.general;
message Error {
string title = 1;
string message = 2;
string stack = 3;
}
message ErrorList {
repeated namespace.general.Error elements = 1;
}
… but the package and import name has been adjusted to match the python directory structure
$ cat ./example_project/src/py/namespace/proto/v1/general/error.proto
syntax = "proto3";
package namespace.proto.v1.general;
message Error {
string title = 1;
string message = 2;
string stack = 3;
}
message ErrorList {
repeated namespace.proto.v1.general.Error elements = 1;
}
Note
The CLI tools and setuptools commands can be used to quickly adjust .proto files
to generate more complex python package structures primarily during development, when the structure is not yet fixed.
To avoid using different versions of .proto sources just generate your .proto files once when
you are ready. The codestare-msg-compiler package only touches package and import
statements in your .proto sources.