User guide¶

The following will outline the capabilities on using releng-tool with an already defined project. For details on building a releng-tool project, see the developer’s guide.

Getting started¶

Depending on the host and how releng-tool has been installed, the tool can be either executed using the call releng-tool (if supported) or explicitly through a Python invoke python -m releng_tool. This guide will assume the former option is available for use. If the alias command is not available on the host system, the latter call can be used instead.

A releng-tool project will be defined by a releng configuration file along with one or more packages found inside a package/ folder. This location can be referred to as the “root directory”. When invoking releng-tool, the tool will look in the current working directory for project information to process. For example, if a project found inside my-project with the single package package-a defined, the following output may be observed:

$ cd my-project
$ releng-tool
extracting package-a...
patching package-a...
configuring package-a...
building package-a...
installing package-a...
generating license information...
(success) completed (0:01:30)

On a successful execution, it is most likely that the releng-tool process will have an asset (or multiple) generated into a images/ location; however, it is up to the developer of a releng-tool project to decide where generated files will be stored.

If a user wishes to pass the directory of a project location via command line, the argument --root-dir can be used:

releng-tool --root-dir my-project/

For a complete list of actions and other argument options provided by the tool, the --help option can be used to show this information:

releng-tool --help

Arguments¶

The command line can be used to specify a single action to perform or provide various options to configure the releng-tool process. Options can be provided before or after an action (if an explicit action is provided). By default, if a user does not specify an action, it is assumed that all steps are to be performed. An example of a user invoking a clean action is as follows:

releng-tool clean

The following outlines available actions:

`clean`	Clean (removes) a series of folders holding content such as extracted archives, built libraries and more. Images and downloaded assets/cache are not removed (see `mrproper` for a more through all cleaning operation). This clean operation will remove files based off the configured output directory. If an output directory is provided (i.e. `--out-dir <dir>`) during a clean event, select folders inside this directory will be removed instead of the output directory (if any) found in the root directory.
`distclean`	Perform a more extreme pristine clean of the releng-tool project. This request removes the `cache/`, `dl/` and `output/` directories found in the root directory or overridden by respective arguments, as well as any mode file flags which may be set. See also the `clean` or `mrproper` actions.
`extract`	All packages will be processed up to the extraction phase (inclusive).
`fetch`	All packages will be processed up to the fetch phase (inclusive; see also offline builds).
`init`	Initialize an empty root directory with a sample project.
`licenses`	A request to generate all license information for the project. Note that license information requires acquiring license documents from packages. Therefore, packages will be fetched/extracted if not already done.
`mrproper`	Perform a pristine clean of the releng-tool project. This request removes the `output/` directory found in the root directory or overridden by the `--out-dir` argument, as well as any mode file flags which may be set. The `cache/` and `dl/` directories will remain untouched. See also the `clean` or `distclean` actions.
`patch`	All packages will be processed up to the patch phase (inclusive).
`sbom`	A request to generate a software build of materials (SBOM) for the project. By default, a releng-tool run will generate an SBOM file at the end of a run. This action can be used to generate an SBOM without requiring a build.
`<pkg>-build`	Performs the build stage for the package. On success, the specified package will have completed its build. If a package has any package dependencies, these dependencies will be processed before the specified package. If the provided package name does not exist, a notification will be generated.
`<pkg>-clean`	Cleans the build directory for package (if it exists). See also the `<pkg>-distclean` action.
`<pkg>-configure`	Performs the configure stage for the package. On success, the specified package will have completed its configuration stage. If a package has any package dependencies, these dependencies will be processed before the specified package. If the provided package name does not exist, a notification will be generated.
`<pkg>-distclean`	Perform a pristine clean of a releng-tool package. This request not only removes the build directory but also any cached file or directory associated with the package. See also the `<pkg>-clean` action.
`<pkg>-exec "<cmd>"`	Invokes a provided command in the package’s build output directory. This package action can be useful for developers attempting to develop/debug a specific package, allowing an easy way to issue commands in a package’s directory without having to manually venture to a package’s output directory. Packages will need to be processed to at least the patch stage before a provided command is issued. An example is as follows: releng-tool libfoo-exec "mycmd arg1 arg2" Package environment variables will be available for the invoked command.
`<pkg>-extract`	Performs the extraction stage for the package. On success, the specified package will have completed its extraction stage. If the provided package name does not exist, a notification will be generated.
`<pkg>-fetch`	Performs the fetch stage for the package. On success, the specified package stage will have completed its fetch stage. If the provided package name does not exist, a notification will be generated.
`<pkg>-install`	Performs the installation stage for the package. On success, the specified package will have completed its installation stage. If a package has any package dependencies, these dependencies will be processed before the specified package. If the provided package name does not exist, a notification will be generated.
`<pkg>-license`	A request to generate the license information for a specific package in a project. Note that license information requires acquiring license documents from the package itself. Therefore, the package will be fetched/extracted if not already done.
`<pkg>-patch`	Performs the patch stage for the package. On success, the specified package will have completed its patch stage. If the provided package name does not exist, a notification will be generated.
`<pkg>-rebuild`	Force a rebuild of a specific package. Once a package has been built, the package will not attempt to be built again. Invoking a rebuild request will tell releng-tool to re-invoke the build step again. This can be useful during times of development where a developer attempts to change a package definition or sources between build attempts. After completing a rebuild, releng-tool will perform the remaining stages of the package (i.e. the installation phase). Users wishing to perform only the rebuild stage are recommended to use `<pkg>-rebuild-only` instead. If using this action, ensure understanding rebuilds has been read to understand this action’s effect.
`<pkg>-rebuild-only`	Force a rebuild of a specific package. Once a package has been built, the package will not attempt to be built again. Invoking a rebuild request will tell releng-tool to re-invoke the build step again. This can be useful during times of development where a developer attempts to change a package definition or sources between build attempts. After completing a rebuild, releng-tool will stop and perform no other changes. Users wishing to perform a rebuild to the installation phase are recommended to use `<pkg>-rebuild` instead. If using this action, ensure understanding rebuilds has been read to understand this action’s effect.
`<pkg>-reconfigure`	Force a re-configuration of a specific package. Once a package has been configured, the package will not attempt to configure it again. Invoking a re-configuration request will tell releng-tool to re-invoke the configuration step again. This can be useful during times of development where a developer attempts to change a package definition or sources between configuration attempts. After completing a re-configuration, releng-tool will perform the remaining stages of the package (i.e. all the way to the installation phase). Users wishing to perform only the re-configuration stage are recommended to use `<pkg>-reconfigure-only` instead. If using this action, ensure understanding rebuilds has been read to understand this action’s effect.
`<pkg>-reconfigure-only`	Force a re-configuration of a specific package. Once a package has been configured, the package will not attempt to configure it again. Invoking a re-configuration request will tell releng-tool to re-invoke the configuration step again. This can be useful during times of development where a developer attempts to change a package definition or sources between configuration attempts. After completing a re-configuration, releng-tool will stop and perform no other changes. Users wishing to perform a re-configuration to the installation phase are recommended to use `<pkg>-reconfigure` instead. If using this action, ensure understanding rebuilds has been read to understand this action’s effect.
`<pkg>-reinstall`	Force a re-installation of a specific package. Once a package has been installed, the package will not attempt to install it again. Invoking a re-installation request will tell releng-tool to re-invoke the installation step again. This can be useful during times of development where a developer attempts to change a package definition or sources between installation attempts. If using this action, ensure understanding rebuilds has been read to understand this action’s effect.

The following outlines available options:

--assets-dir <dir>

Directory to hold cache and download folders instead of using a configured root directory.

This argument can be ignored if --cache-dir or --dl-dir are set.

--cache-dir <dir>

Directory for distributed version control cache information (defaults to <root>/cache).

--config <file>

Configuration file to load (default: <root>/releng).

--debug

Show debug-related messages.

-D, --development [<mode>]

Enables development mode.

--dl-dir <dir>

Directory for download archives (defaults to <root>/dl).

-F, --force

Triggers a forced request for the releng-tool invoke. This entails:

Packages will be processed as if a re-configuration request has been made.
If an explicit fetch request is made (fetch or <pkg>-fetch), any packages which cache to a file will have their cache files deleted to be re-fetched.

-h, --help

Show this help.

--images-dir <dir>

Directory for image outputs (defaults to <root>/output/images).

-j, --jobs <jobs>

Numbers of jobs to handle (default to 0; automatic).

-L, --local-sources [[<pkg>@]<dir>]

Enables local-sources mode.

Without a directory provided, sources of internal packages will be looked for in the parent directory of the configured root directory. Users may use this argument multiple times to override the local-sources configuration. If a package-specific override is provided (a value prefixed with a package’s name with an at sign (@)), sources for that package will be looked for inside the provided path.

--nocolorout

Explicitly disable colorized output.

--out-dir <dir>

Directory for output (builds, images, etc.; defaults to <root>/output).

--root-dir <dir>

Directory to process a releng-tool project (defaults to the working directory).

--sbom-format <fmt>

The format to use when generating a software build of materials (SBOM). Multiple formats can be provided (comma-separated).

Type	Value
CSV	`csv`
HTML	`html`
JSON	`json`
Text	`text` (default)
XML	`xml`

Understanding rebuilds¶

As packages are processed in order (based off of detected dependencies, if any), each package will go through their respective stages: fetching, extraction, patching, configuration, building and installation. While a package may not take advantage of each stage, the releng-tool will step through each stage to track its progress. Due to the vast number of ways a package can be defined, the ability for releng-tool to determine when a previously executed stage is “stale” is non-trivial. Instead of attempting to manage “stale” package stages, releng-tool leaves the responsibility to the builder to deal with these scenarios. This idea is important for developers to understand how it is possible to perform rebuilds of packages to avoid a full rebuild of the entire project.

Consider the following example: a project has three packages module-a, module-b and module-c which are C++-based. For this example, project module-b depends on module-a and project module-c depends on module-b; therefore, releng-tool will process packages in the order module-a -> module-b -> module-c. In this example, the project is building until a failure is detected in package module-c:

$ releng-tool
[module-a built]
[module-b built]
[error in module-c]

A developer notices that it is due to an issue found in module-b; however, instead of attempting to redo everything from a fresh start, the developer wishes to test the process by manually making the change in module-b to complete the build process. The developer makes the change, re-invokes releng-tool but still notices the build error occurs:

$ releng-tool
[error in module-c]

The issue here is that since module-b has already been processed, none of the interim changes made will be available for module-c to use. To take advantage of the new implementation in module-b, the builder can signal for the updated package to be rebuilt:

$ releng-tool module-b-rebuild
[module-b rebuilt]

With module-b in a more desired state, a re-invoke of releng-tool could allow module-c to be built.

$ releng-tool
[module-c built]

This is a very simple example to consider, and attempts to rebuild can vary based on the packages, changes and languages used.

Tips¶

Offline builds¶

A user can prepare for an offline build by using the fetch action:

releng-tool fetch

Package content will be downloaded into the respective dl/ or cache/ folders. Future builds for the project should no longer need external access until these folders are removed.

Parallel builds¶

A stage for a package (such as a build stage) can take advantage of multiple cores to perform the step. By default, releng-tool will attempt to run as many jobs for a stage equal to the amount of physical cores on the host system. The amount of jobs available for a stage can be configured using the --jobs argument. For example, if a user wishes to override the amount of jobs attempted for stages to two jobs, the following can be used:

releng-tool --jobs 2

Note that a developer may restrict the amount of jobs allowed for a specific package if a package cannot support parallel processing.

Privileged builds¶

It is never recommended to invoke a build with elevated (e.g. root) privileges. A builder invoking in an elevated environment runs the risk of a misconfigured releng-tool project dirtying or destroying the builder’s host environment.

License generation¶

At the end of a releng-tool invoke, the final stages will compile a list of package license information (if licenses are defined). If a user wishes to compile a project’s list of license information without performing an entire build, the licenses action can be used:

releng-tool licenses

License information can then be found in the root directory’s <root>/licenses folder.

Advanced builder capabilities¶

Development mode¶

Development mode provides a way for a user to request to process packages against development versions of sources rather than using fixed versions. A package will typically target a stable release, either pointing to a specific archive to download or a specific tag to clone from. However, for some builds, a user may wish to build a specific package against their main development branch (e.g. the main branch of a Git repository) or a long-term stable release branch. Packages can be defined to target these specific revisions if running in development mode.

To enable development mode, invoking releng-tool with the --development argument will enable the mode. Future calls to releng-tool for the project will use a development revision for packages where appropriate. For example:

$ releng-tool --development [<mode>]
(success) configured root for development mode
$ releng-tool
~building against development sources~
...

Development mode is persisted through the use of a file flag in the root directory.

Consider the following example: a package defines multiple revisions to fetch sources from:

LIBFOO_SITE = 'https://example.com/libfoo.git'
LIBFOO_REVISION = {
    DEFAULT_REVISION: '1.2',
    'develop': 'main',
    'lts': '1.1.x',
}

A build would normally use the 1.2 tag for this package. However, if an environment is configured to use the develop development mode:

$ releng-tool --development develop

This package would use the main branch instead.

Projects can also target specific sites based off the development mode. This can be useful if a package uses a built archive for a stable release, but having development sources fetched from a repository. For example:

LIBFOO_SITE = {
    DEFAULT_SITE: 'https://pkgs.example.com/releases/libfoo-${LIBFOO_VERSION}.tar.gz',
    'test': 'https://git.example.com/libfoo.git',
}

LIBFOO_REVISION = {
    'test': 'main',
}

In a normal execution, a tar.gz archive would be downloaded for the package. However, if an environment is configured to use the test development mode, sources will be fetched from the Git repository on the main branch.

Simple development modes are also supported. Packages can use the LIBFOO_DEVMODE_REVISION option to hint at a development revision to pull.

LIBFOO_DEVMODE_REVISION = 'main'
LIBFOO_REVISION = 'v3.0'

A build would normally use the v3.0 tag for this package. However, if an environment is configured a non-explicit development mode:

$ releng-tool --development

This package would use the main branch instead.

A user can either disable development mode by performing a mrproper or can manually remove the file flag.

Local-sources mode¶

Note

Clean events (such as releng-tool clean) will not touch packages using sources found alongside the output directory

Local-sources mode provides a way for a developer to build internal-flagged packages using sources found alongside the root directory (or a specific provided directory), instead of having releng-tool attempt to fetch them from remote instances. This is primarily for developers who desire to manually manage source content outside the releng-tool environment. Local-sources mode only works for internally flagged packaged. Consider the following example: a releng-tool project has a package called liba. When releng-tool is invoked in normal configurations, the package will do fetching, extraction and patching to prepare the directory <root>/output/build/liba-<version>. However, if a builder has configured the working root for local-sources mode, sources for liba will be used from the folder <root>/../liba instead.

When in local-sources mode, an internal package will skip the fetching, extraction and patching stages in order to prevent undesired manipulation of developer-prepared sources. Another consideration to note is the use of clean operators while in local-sources mode. Continuing with the above example, if a user invokes releng-tool liba-clean, the operation will not remove the <root>/../liba folder. Responsibility to managing a clean liba package will be left with the user.

To enable local-sources mode, invoking releng-tool with the --local-sources argument will enable the mode. Future calls to releng-tool for the project will use local sources for packages defined as internal packages. For example:

$ releng-tool --local-sources
(*) <parent>
(success) configured root for local-sources mode
$ releng-tool
~building against local sources~
...

Local-sources mode is persisted through the use of a file flag in the root directory.

If a user provides a directory for the --local-sources argument, packages will be looked for in the provided folder instead of the parent of the configured root directory. For example:

$ releng-tool --local-sources ~/workdir
(*) ~/workdir
(success) configured root for local-sources mode
$ releng-tool
~building against local sources~
...

In the above example, if a project had an internal package liba, sources for liba will be used from the folder ~/workdir/liba.

Users can also provide package-specific overrides. If a user provides a path which is prefixed with a package’s name and at sign (@), the sources for the provided package will be used from the respective folder:

$ releng-tool -L ~/workdir
$ releng-tool -L libb@/mnt/sources/libb
$ releng-tool -L libc@
(*) ~/workdir
(libb) /mnt/sources/libb
(libc) <unset>
(success) configured root for local-sources mode
$ releng-tool
~building against local sources~
...

In the above example, if a project had internal packages liba, libb and libc, the following paths will be used:

Sources for liba will be used from the folder ~/workdir/liba,
Sources for libb will be used from the folder /mnt/sources/libb; and,
Sources for libc will not be fetched locally.

A user can either disable local sources mode by performing a mrproper or can manually remove the file flag.

Configuration overrides¶

If a builder needs to (for example) override a tool location or package site, a user can define either environment options or setup a configuration override script releng-overrides. It is never recommended to persist a configuration overrides file into a project’s source repository.

Extraction tool overrides¶

The override_extract_tools option inside a configuration override script allows a dictionary to be provided to map an extension type to an external tool to indicate which tool should be used for extraction. For example, when a .zip archive is being processed for extraction, releng-tool will internally extract the archive; however, a user may wish to override this tool with their own extraction utility. Consider the following example:

override_extract_tools = {
    'zip': '/opt/my-custom-unzip {file} {dir}',
}

The {file} key will be replaced with the file to be extracted, and the {dir} key will be replaced where the contents should extract to.

Revision overrides¶

The override_revisions option inside a configuration override script allows a dictionary to be provided to map a package name to a new revision value. Consider the following example: a project defines module-a and module-b packages with package module-b depending on package module-a. A developer may be attempting to tweak package module-b on the fly to test a new capabilities against the current stable version of module-a; however, the developer does not want to explicitly change the revision inside package module-b’s definition. To avoid this, an override can be used instead:

override_revisions = {
    'module-b': '<test-branch>',
}

The above example shows that package module-b will fetch using a test branch instead of what is defined in the actual package definition.

Site overrides¶

The override_sites option inside a configuration override script allows a dictionary to be provided to map a package name to a new site value. There may be times where a host may not have access to a specific package site. To have a host to use a mirror location without having to adjust the package definition, the site override option can be used. For example, consider a package pulls from site git@example.com:myproject.git; however, the host example.com cannot be access from the host machine. If a mirror location has been setup at git@example.org:myproject.git, the following override can be used:

override_sites = {
    '<package-name>': 'git@example.org:myproject.git',
}

Tool overrides¶

Environment variables can be used to help override external tool invoked by the releng-tool process. For example, when invoking CMake-based projects, the tool cmake will be invoked; however, if a builder is running on CentOS and CMake v3.x is desired, the tool cmake3 needs to be invoked instead. To configure this, an environment variable can be set to switch which tool to invoke. Consider the following example:

$ export RELENG_CMAKE=cmake3
$ releng-tool
[cmake3 will be used for cmake projects]

Quirks¶

releng-tool also provides a series of configuration quirks to deal with rare host environment scenarios where releng-tool may be experiencing issues. See configuration quirks for more information.