codestare.compiletools package

codestare.compiletools.has_module(*modules: str)

Try importing module, catch ImportError

Parameters

*modules – module names

Returns: True if module is importable else False

codestare.compiletools.find_proto_files(*paths: pathlib.Path, recursive: bool = True) → List[pathlib.Path]

Return relative paths of .proto files in paths

Parameters

• *paths – We glob() the files in the paths

• recursive – check recursively in all paths if True

Returns

List of unique paths of found .proto files, relative to search path

Submodules

codestare.compiletools.cmd module

Googles own python plugin implementation for protoc does not compile files with relative imports, like e.g. betterproto (related issue). Several third party tools try to “fix” this by either pre- or post-processing the generated files, or using a custom protoc plugin (like betterproto).

Conversely, betterproto’s relative imports only succeed when all sources are compiled simultaneously (all proto files as arguments to protoc invocation, otherwise files might be overwritten). This would break support for build tools like cmake / make, allowing parallel compilation of .proto files i.e. it seems not possible to implement in the official plugin.

Google itself develops an alternative protobuf module which aims to use protocol buffers with more idiomatic python code but has no official compiler plugin as of now. A third party plugin plusplugin loosely based on the mypy plugin is available (experimental).

codestare.compiletools.cmd.compare_files()

Context manager to monkey patch distutils.file_util.copy_file() to only copy files which are not equal according to filecmp.cmp().

Yields

None

Example

Use the context manager with methods implicitly using distutils.file_util.copy_file():

class CustomCommand(distutils.cmd.Command):
    def copy_not_equal(new, old):
        with compare_files():
            self.copy_tree(new, old)

codestare.compiletools.cmd.temp_dir()

Context manager to redirect build directory to temporary directory. This is the easiest way to only “build” changed protobuf modules: We build to temp dir, then copy to real build dir if files are different.

Note

There is no way to tell the protoc compiler to only build files that have changed / updated sources. This is typically the task of make or some other build script, and all files passed to protoc will be built (in theory the plugin could also handle this, but none of the default plugins do). This means they will have new file generation timestamps and will

• probably be marked as changed in the version control system

• be copied to the build directory by the python build toolchain

Because of this, we need to compare the file contents instead of the timestamps, even when building to a temporary directory, see compare_files().

Yields

pathlib.Path – new build dir

class codestare.compiletools.cmd.PathCommand(dist)

Bases: distutils.cmd.Command, abc.ABC

Abstract command class supporting verification of options representing path and directory lists

ensure_path_list(option)

Ensure option is list of paths

Parameters

option (str) – name of option

ensure_dir_list(option)

Ensure option is list of directories

Parameters

option (str) – name of option

class codestare.compiletools.cmd.CompileBase(dist)

Bases: codestare.compiletools.cmd.PathCommand

Base class for setuptools commands handling compilation of .proto files.

includes

Directories containing .proto files to pass as includes to protoc

Type

List[pathlib.Path]

output

compilation output directory

Type

pathlib.Path

files

actual list of .proto files to compile

Type

List[pathlib.Path], optional

proto_package

if files is missing, search this path for .proto files

Type

pathlib.Path, optional

force

Force compilation / copying of all generated files disregarding changed contents. Defaults to False.

Type

bool

dry_run

Don’t copy generated files to build dir, only print list of generated files. Defaults to False.

Type

bool

options

List of option flags to trigger compilation flavors

Type

List[str]

protoc

Path to protoc executable. If not supplied, protoc needs to be in $PATH

Type

pathlib.Path

plugin_params

Some protoc plugins support additional parameters

Type

str, optional

user_options = [('protoc=', None, 'protoc compiler location'), ('output=', None, 'Output directory for compiled files'), ('includes=', None, 'Include directories for .proto files'), ('force', 'f', 'forcibly build everything (ignore file timestamps)'), ('files=', None, 'Protobuf source files to compile'), ('plugin_params=', None, 'parameters passed to protoc plugin'), ('options=', 'o', 'Options for compilation, possible values are [<CompileOption.JAVA: 1>, <CompileOption.JSLIBRARY: 2>, <CompileOption.JSINDIVIDUAL: 4>, <CompileOption.CSHARP: 8>, <CompileOption.CPP: 16>, <CompileOption.PYTHON_BETTER_PROTO: 32>, <CompileOption.PYTHON_MYPY: 64>, <CompileOption.PYTHON_PROTOPLUS: 128>, <CompileOption.PYTHON_BASIC: 256>] (default)')]

User options for distutils.cmd.Command. See CompileBase.run() for more info about how they are used in particular.

Each option is used to supply the corresponding attribute: for more information about e.g. user_options['force'] refer to force.

User options are passed between build steps. If not supplied specifically for this build command, the following mapping occurs (options specified multiple times will take the first existing default value):

option	default {command_name}.{option_name}
includes	rewrite_proto.outputs
output	compile_proto.build_lib
includes	compile_proto.include_proto
proto_package	compile_proto.proto_package
force	compile_proto.force
dry_run	compile_proto.force

run()

Calls Compiler.compile() with user_options as keyword arguments.

This triggers a compilation of files using the protoc executable, passing includes as includes (-I flag) and plugin_params, using the plugin and default options for options (see :mod:codestare.compiletools.options) with output directory output.

If force is not set, only generated files that don’t exist with the same contents in output will be copied from the temporary build directory.

If dry_run is set, no files will be copied whatsoever, and only the protoc invocation will be shown.

class codestare.compiletools.cmd.CompileProtoPython(dist)

Bases: codestare.compiletools.cmd.CompileBase

Command to compile with compilation option PYTHON_BASIC and generate __init__.py files with GenerateInits

class codestare.compiletools.cmd.CompileProtoMypy(dist)

Bases: codestare.compiletools.cmd.CompileBase

Command to compile with compilation option PYTHON_MYPY, to generate .pyi stubs.

class codestare.compiletools.cmd.CompileBetterproto(dist)

Bases: codestare.compiletools.cmd.CompileBase

Command to compile with compilation option PYTHON_BETTER_PROTO and generate __init__.py files with GenerateInits

class codestare.compiletools.cmd.CompileProtoPlus(dist)

Bases: codestare.compiletools.cmd.CompileBase

Command to compile with compilation option PYTHON_BETTER_PROTO and generate __init__.py files with GenerateInits

Note

Although generating __init__.py files via the GenerateInits setuptools command is supported for CompileProtoPlus, the protoc plugin used in this compilation handles generation of init files.

It is recommended to not use the generate_inits command or at least not to use force to avoid overwriting the files.

class codestare.compiletools.cmd.CompileProto(dist)

Bases: codestare.compiletools.cmd.PathCommand

Triggers compilation of all available python flavors that are turned on via flavor. If proto_package and include_proto are supplied, enforce package structure according to proto_package by rewriting included source files with rewrite subcommand

include_proto

root dir for proto files

Type

List[pathlib.Path]

proto_package

parent package that will be enforced for protobuf modules

Type

str

build_lib

output directory for protobuf library

Type

os.path.PathLike

flavor

flavor of compiled code

Type

str, optional

force

forcibly build everything (ignore file timestamps)

Type

bool, optional

dry_run

don’t do anything but show protoc commands

Type

bool, optional

clean

remove rewritten *.proto files after compilation

Type

bool, optional

class Flavor(value)

Bases: enum.Enum

Possible Flavors for python compilations

flavors = {'better': Flavor.BETTER, 'mypy': Flavor.MYPY, 'plus': Flavor.PLUS, 'python': Flavor.BASIC}

Mapping of options to enums

user_options = [('include-proto', None, 'root dir for proto files'), ('proto-package', None, 'parent package that will be enforced for protobuf modules'), ('build-lib', None, 'output directory for protobuf library'), ('flavor', None, 'flavor of compiled code, one of mypy,python,better,plus'), ('force', 'f', 'forcibly build everything (ignore file timestamps)'), ('dry-run', None, "don't do anything but show protoc commands")]

User options for distutils.cmd.Command behaviour.

Each option is used to supply the corresponding attribute: for more information about e.g. user_options['force'] refer to force.

User options are passed between build steps. If not supplied specifically for this build command, the following mapping occurs (options specified multiple times will take the first existing default value):

option	default {command_name}.{option_name}
build_lib	build_py_proto.build_lib
include_proto	build_py_proto.include_proto
force	build_py_proto.force
dry_run	build_py_proto.dry_run
keep_proto_files	build_py_proto.keep_proto_files

run()

Trigger all compilations and add compiled packages to distribution if missing.

Note

If python packages are generated which are not already part of the distribution (through setting packages option in setup.py or setup.cfg as usual) this command will issue a warning

class codestare.compiletools.cmd.CleanProto(dist)

Bases: codestare.compiletools.cmd.PathCommand

Setuptools command to clean .proto sources after compilation. Respects dry-run setting from compile_proto.

Attributes: clean_dirs (List[str]): directories where all *.proto files will be removed recursive (bool): also remove files in sub-directories of clean_dir [default] dry_run (bool): fake clean only print paths

class codestare.compiletools.cmd.RewriteProto(dist)

Bases: codestare.compiletools.cmd.PathCommand

Setuptools command to rewrite .proto sources in a way that produces a specified python package structure. Respects dry-run setting from compile_proto.

proto_package

parent package that will be enforced for protobuf modules

Type

str

inplace

write output to compile_proto include directory

Type

bool

use_build

write output to build_lib directory [default]

Type

bool

dry_run

fake write only print paths

Type

bool

class codestare.compiletools.cmd.GenerateInits(dist)

Bases: codestare.compiletools.cmd.PathCommand

Setuptools command to generate missing __init__.py files in generated python package tree.

run() → None

Generate recursive init files with wildcard imports for a package.

codestare.compiletools.compile module

Protoc compiler wrapper CLI (using fire)

class codestare.compiletools.compile.Compiler(protoc=distutils.spawn.find_executable('protoc'))

Bases: object

Wrapper around protobuf compiler.

See compile-proto OPTIONS for all possible compilation options. See compile-proto (no args) and compile-proto --help for more help

OPTIONS = ['[java] java', '[cs] csharp', '[cpp] cpp', '[better] python with `better_proto` plugin', '[mypy] mypy python stubs', '[plus] python_protoplus', '[py] python', '[js] javascript as library | javascript individual', '[all] java | javascript as library | javascript individual | csharp | cpp | python with `better_proto` plugin | mypy python stubs | python_protoplus | python']

Supported compile options, multiple options may be specified.

call(*proto_files, includes: Optional[str] = None, protohelp=False, dry_run=False, **options)

Just a wrapper around the protoc compiler.

See compile-proto call --help to print help of protoc command See compile-proto call -- --help for additonal help with the call command See compile-proto protoc for the path of the used protoc compiler executable

Parameters

• includes – Directories to use as includes

• protohelp – Use this flag to pass --help to protoc invocation

• options – Will be passed as flags to the protoc command (only --flag syntax supported, no single dash)

• dry_run – If True, don’t do anything except printing the command

compile(*protoc_files, options=_DEFAULT, quiet=False, plugin_params: str = '', output=os.getcwd(), **kwargs) → Optional[str]

Compile for given options, see compile-proto compile -- --help

Parameters

• quiet – Don’t print output from protoc plugin if possible

• output – output directory (default: working directory)

• options – one or multiple options see compile-proto OPTIONS default: [py]

• protoc_files – files to compile, passed through to protoc

• plugin_params – mapping of additional parameters for the protoc plugin (not the compiler itself)

• kwargs – Passed to protoc invocation, see compile-proto call – –help.

Returns

the output path

class codestare.compiletools.compile.Rewriter(root_package: str, output_root: os.PathLike = './')

Bases: object

Rewrite proto files to force python package structure which mirrors protobuf package declarations.

read(*dirs: os.PathLike)

Reads .proto files from directories, manipulate them with the other commands afterwards.

property root_package

root package for generated directory structure

Returns

root package which is enforced

Return type

str

property output_root

Root directory of new directory structure.

property calculated_packages

Returns

Dictionary mapping file contents to package names

Return type

dict

fix_imports()

Modify internal file content representation to match internal file locations, so that imports are importing the right files.

Returns

Reference to self to chain commands with fire

Return type

Rewriter

fix_packages()

Modify internal content representation such that package declarations in .proto source files match the internal directory structure

Returns

Reference to self to chain commands with fire

Return type

Rewriter

content(filename: os.PathLike) → str

View the internal file representations

Parameters

filename – Input file path

Returns

contents of internal file representation

Return type

str

write(dry_run=True) → List[str]

Write internal content representation to output_root according to internal package mapping

Parameters

dry_run (bool) – if True don’t actually write outputs.

Returns

Reference to self to chain commands with fire

Return type

Rewriter

codestare.compiletools.compile.compile_proto()

Entry point for CLI.

fire Compiler

codestare.compiletools.compile.rewrite_proto()

Entry point for CLI

fire Rewriter

codestare.compiletools.options module

Options for compilation are represented using CompileOption

class codestare.compiletools.options.CompileOption(value)

Bases: enum.Flag

Describes available compile options

JAVA = 1

JSLIBRARY = 2

JSINDIVIDUAL = 4

CSHARP = 8

CPP = 16

PYTHON_BETTER_PROTO = 32

PYTHON_MYPY = 64

PYTHON_PROTOPLUS = 128

PYTHON_BASIC = 256

JAVASCRIPT = 6

PYTHON = 480

ALL = 511

classmethod from_str(value) → codestare.compiletools.options.CompileOption

Convert a string to an enum value

Parameters

value (str) – valid values are defined by arguments

Returns

Enum flag

Return type

CompileOption

classmethod from_string_list(values)

Convert a string list to an enum value (combined flag for all values in list)

Parameters

values (List[str]) – list of values according to arguments

Returns

combined flag

Return type

CompileOption

property arguments: List

Arguments that can be used to trigger this compilation

property disjunct: List[codestare.compiletools.options.CompileOption]

Return all options which are not combinations of other options

property is_composite

Like normal integer flags, individual flags are all powers of 2, see https://docs.python.org/3/library/enum.html#flag combined flags are not, so we test that with o.value & (o.value - 1)

parameters(*args, **kwargs)

some options can use request parameters for the plugin, this is e.g. how the javascript plugin implements “library / binary” compilation.

• JSLIBRARY adds library='protobuf_library',binary

• JSINDIVIDUAL adds import_style='commonjs',binary

• PYTHON_PROTOPLUS adds readable_imports