Advanced Usage¶

This section covers advanced usage of the CLI tools compile-proto and rewrite-proto you need to install the [cli] extra to use them.
The CLI tools are build using fire. Refer to the fire documentation, especially the fire guide section about chaining commands for more information.
The examples on this page use the same example project as the Setup tutorial.

Note

The example project is available for download. You might want to test the behaviour of the build commands on some files that you don’t care about ;)

$ tree ./example_project
./example_project
├── build
├── pyproject.toml
├── setup.cfg
├── setup.py
└── src
    ├── proto
    │   ├── general
    │   │   ├── error.proto
    │   │   └── success.proto
    │   └── services
    │       ├── reply.proto
    │       ├── request
    │       │   └── subscription.proto
    │       ├── request.proto
    │       └── service.proto
    └── py
        └── namespace
            └── proto
                ├── __init__.py
                ├── additonal_module.py
                └── v1

10 directories, 11 files

example_project/src/proto/general/error.proto¶

syntax = "proto3";
package namespace.general;

message Error {
    string title = 1;
    string message = 2;
    string stack = 3;
}

message ErrorList {
    repeated namespace.general.Error elements = 1;
}

As you can see, the example project .proto files use package declaration starting with “namespace”. Since they are located inside the src/proto directory compiling them with protoc and the default python plugin would produce a python package named “proto” or “src.proto”, since the default python plugin does not care about the package declaration in the .proto files and instead creates packages according to the directory structure.

rewrite-proto¶

Root Package¶

You need to set a root package. The package can be any string that is a valid python package name (related PEP). In our Setup tutorial we wanted the package to be “namespace.proto.v1”. If you don’t set a root package, you will get an error:

$ rewrite-proto - read ./example_project/src/proto
ERROR: The function received no value for the required argument: root_package
Usage: rewrite-proto --root_package=ROOT_PACKAGE <flags>
  optional flags:        --output_root

For detailed information on this command, run:
  rewrite-proto --help

You can inspect how the rewrite command would process the packages like so:

$ rewrite-proto --root_package "foo.bar"
- read ./example_project/src/proto
- calculated_packages
example_project/src/proto/services/service.proto:              foo.bar.namespace.services
example_project/src/proto/services/request.proto:              foo.bar.namespace.services
example_project/src/proto/services/reply.proto:                foo.bar.namespace.services
example_project/src/proto/services/request/subscription.proto: foo.bar.namespace.services.request
example_project/src/proto/general/error.proto:                 foo.bar.namespace.general
example_project/src/proto/general/success.proto:               foo.bar.namespace.general

Here fire allows us to chain the commands. We first read() the .proto files from the src/proto directory, and then inspect the calculated_packages attribute.

Note

As you can see, the new root package gets prepended, the namespace package is retained. If your root package ends with some prefix of the protobuf package declaration only the excessive part is prepended, as you can see here:

$ rewrite-proto --root_package "foo.namespace"
- read ./example_project/src/proto
- calculated_packages
example_project/src/proto/services/service.proto:              foo.namespace.services
example_project/src/proto/services/request.proto:              foo.namespace.services
example_project/src/proto/services/reply.proto:                foo.namespace.services
example_project/src/proto/services/request/subscription.proto: foo.namespace.services.request
example_project/src/proto/general/error.proto:                 foo.namespace.general
example_project/src/proto/general/success.proto:               foo.namespace.general

Reading Files¶

The read() method recursively searches passed directories for *.proto files. It will raise an error if you pass a directory that does not contain *.proto files. You can pass multiple directories

$ rewrite-proto --root_package "foo"
- read ./example_project/src/proto/general ./example_project/src/proto/services/request
- calculated_packages
example_project/src/proto/general/error.proto:                 foo.namespace.general
example_project/src/proto/general/success.proto:               foo.namespace.general
example_project/src/proto/services/request/subscription.proto: foo.namespace.services.request

Fixing Imports and Packages¶

The purpose of the rewrite-proto tool is to process the input .proto files so that the declared packages match the enforced directory structure and imports are preserved. For this and internal mapping of imports and filenames is created once files are read(). After calling fix_imports() or fix_packages() you may inspect the internal representation via content().

$ rewrite-proto --root_package "foo"
- read ./example_project/src/proto
- fix_packages
- content example_project/src/proto/services/request/subscription.proto
syntax = "proto3";
package foo.namespace.services.request;

message TopicSubscription {
    string client_id = 1;
    repeated string subscribe_topics = 2;
    repeated string unsubscribe_topics = 3;
}

message RegexSubscription {
    string client_id = 1;
    repeated string subscribe_regex = 2;
    repeated string unsubscribe_regex = 3;
}

As you can see, the package has been changed.

$ rewrite-proto --root_package "foo"
- read ./example_project/src/proto
- fix_packages
- content example_project/src/proto/services/request.proto
syntax = "proto3";
package foo.namespace.services;

import "services/request/subscription.proto";

message ServiceRequest {
    string topic = 1;
    oneof type {
        foo.namespace.services.request.TopicSubscription topic_subscription = 2;
        foo.namespace.services.request.RegexSubscription regex_subscription = 3;
    }
}

Here the package of the imported message types has been changed to match the changed package in their .proto declaration, but the import statement is now wrong. Since the processed example_project/src/proto/services/request/subscription.proto file will be put into a directory [...]/foo/namespace/services/request to create a python package foo.namespace.services.request at the same location (recall that the python package output is only determined by the directory structure of the .proto sources), the import will not find it. To fix this, one also needs to call fix_imports()

$ rewrite-proto --root_package "foo"
- read ./example_project/src/proto
- fix_packages
- fix_imports
- content example_project/src/proto/services/request.proto
syntax = "proto3";
package foo.namespace.services;

import "foo/namespace/services/request/subscription.proto";

message ServiceRequest {
    string topic = 1;
    oneof type {
        foo.namespace.services.request.TopicSubscription topic_subscription = 2;
        foo.namespace.services.request.RegexSubscription regex_subscription = 3;
    }
}

Now the imports match the new directory structure.

Writing new directory structure¶

You can set the output_root, by default the new directory structure will be created in the current working directory.

$ rewrite-proto --root_package "foo" - write --help
INFO: Showing help with the command 'rewrite-proto --root_package foo - write -- --help'.

NAME
    rewrite-proto --root_package foo write - Write internal content representation to :attr:`.output_root` according to internal :attr:`package mapping <.calculated_packages>`

SYNOPSIS
    rewrite-proto --root_package foo - write <flags>

DESCRIPTION
    Write internal content representation to :attr:`.output_root` according to internal :attr:`package mapping <.calculated_packages>`

FLAGS
    --dry_run=DRY_RUN
        Default: True
        if True don't actually write outputs.

Note

As you can see --dry-run is the default. If you actually want to write your files, pass --nodry-run according to fire documentation

$ rewrite-proto --root_package foo --output_root ./example_project/build/
- read ./example_project/src/proto
- fix_packages
- fix_imports
- write --nodry-run
example_project/build/foo/namespace/services/service.proto
example_project/build/foo/namespace/services/request.proto
example_project/build/foo/namespace/services/reply.proto
example_project/build/foo/namespace/services/request/subscription.proto
example_project/build/foo/namespace/general/error.proto
example_project/build/foo/namespace/general/success.proto

$ tree ./example_project/build
./example_project/build
└── foo
    └── namespace
        ├── general
        │   ├── error.proto
        │   └── success.proto
        └── services
            ├── reply.proto
            ├── request
            │   └── subscription.proto
            ├── request.proto
            └── service.proto

5 directories, 6 files

compile-proto¶

This tool is basically just a wrapper around the protoc compiler (which needs to be installed in your $PATH or be passed as a parameter) with nicely defined “compile options” for compilation with different plugins and plugin parameters.

$ compile-proto compile -- --help
NAME
    compile-proto compile - Compile for given options, see ``compile-proto compile -- --help``

SYNOPSIS
    compile-proto compile <flags> [PROTOC_FILES]...

DESCRIPTION
    Compile for given options, see ``compile-proto compile -- --help``

POSITIONAL ARGUMENTS
    PROTOC_FILES
        files to compile, passed through to protoc

FLAGS
    --options=OPTIONS
        one or multiple options see `compile-proto OPTIONS` default: [py]
    --quiet=QUIET
        Don't print output from protoc plugin if possible
    --plugin_params=PLUGIN_PARAMS
        Type: 'str'
        mapping of additional parameters for the protoc plugin (not the compiler itself)
    --output=OUTPUT
        output directory (default: working directory)
    Additional flags are accepted.
        Passed to protoc invocation, see compile-proto call -- --help.

The options are defined in the CompileOption enum. The enum class implements some properties such that every enum value or combination of enum values (the CompileOption enum is a enum.IntFlag, i.e. multiple values can be combined) can be mapped to protoc plugins, parameters, and so on. You can explore this using the interactive mode of the tool.

Each CompileOption which is not "composite" has an associated plugin. Refer to the source code of the CompileOption enum for more info.

Advanced Usage¶

rewrite-proto¶

Root Package¶

Reading Files¶

Fixing Imports and Packages¶

Writing new directory structure¶

compile-proto¶

codestare-msg-compiler

Navigation

Related Topics