Developer guide¶

The following will outline details on how to prepare a releng-tool project to be used for a project. For details on a user interaction with releng-tool, see the user’s guide.

Prelude¶

A releng-tool project can define multiple packages, each can be based off of different languages, configured to use custom toolchains and more. Every package has multiple stages (such as fetching, configuring, building, etc.) which can help contribute to a target sysroot. Once all packages are processed, the target sysroot can be packaged for distribution.

The following outlines the (default) common directory/file locations for a releng-tool project:

cache/ - cached content from select package sources (e.g. dvcs, etc.)
dl/ - archives for select package sources (e.g. .tgz, .zip, etc.)
package/ - container for packages
package/<package>/ - a package-specific folder
package/<package>/<package> - a package definition
output/ - container for all output content
output/build/ - container for package building
output/host/ - area to hold host-specific content
output/images/ - container for final images/archives
output/staging/ - area to hold staged sysroot content
output/target/ - area to hold target sysroot content
releng - project configuration

How these directories and files are used can vary on how a developer defines a releng-tool project. Consider the following example:

releng-tool will load the project’s configuration and respective package definitions to determine what steps need to be performed.
Package sources can be downloaded into either the cache/ or dl/ folder, depending on what type of sources will be fetched. For example, Git sources will be stored inside of the cache/ to take advantage of its distributable nature, and archive files (such as .tgz, .zip, etc.) will be stored inside the dl/ directory.
Each package will be extracted into its own output directory inside output/build/. The working areas for packages allow a package to be patched, configured and built based on how the developer configures the respective packages.
Once packages are built, their final executables, libraries, etc. can be installed into either the host area (output/host/), staging area (output/staging/) or the target area (output/target/) depending on what has been built. The target area is designed for the final set of assets produced from a build; the intent is that the files contained inside this folder are planned to be used on a target system (stripped, cross-compiled, etc.). A staging area is like a target area but may contain more content such as headers not intended for a final target, interim development assets, and more. Host content is designed for content built for the host system which other packages may depend on.
At the end of the releng-tool process, a post-stage script can be invoked to help archive/package content from the target area (output/target/) into the images folder (output/images/). For example, generating an archive output/images/my-awesome-project-v1.0.0.tgz.

Not all projects may use each of these folders or take advantage of each stage. While various capabilities exist, it does not mean releng-tool will handle all the nitty-gritty details to make a proper build of a project. For example:

If a developer wishes to cross-compile a package to a target, they must ensure the package is configured in the proper manner to use a desired toolchain.
If a developer wants to process a Python package, they must ensure the proper interpreter is used if they cannot rely on the host’s default interpreter.
If a developer develops script-based packages, they must ensure that these scripts properly handle multiple re-invokes (e.g. if a builder performs a rebuild on a package).

releng-tool will attempt to provide an easy way to deal with fetching sources, ensuring projects are invoked in order, and more; however, the more advanced features/actions a developer may want in their project (like the examples mentioned above), the more a developer will need to manage their project.

Getting started¶

releng-tool is made on Python. This document will outline package definitions, scripts and more which are most likely Python scripts, and in turn will be invoked/processed by the releng-tool. While releng-tool supports running on various host systems (e.g. Linux, OS X, Windows, etc.), this guide will primarily show examples following a Unix-styled file system.

The following will provide a very simple script-based project solution to get started. This example will make a project with two packages, setup a dependency between them and setup scripts to help show a developer how packages are processed. If there is no desire to make a simple project example, one may venture to the actually getting started section.

To start, make a new folder for the project, two sample packages and move into the root folder:

$ mkdir -p my-project/package/libx
$ mkdir -p my-project/package/liby
$ cd my-project/

Inside the libx project, the package definition and script-based files will be created. First, build the package definition my-project/libx/libx with the following content:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

LIBX_VERSION = '1.0.0'

Next, create a build script for the libx project my-project/libx/libx-build with the following content:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

print('invoked libx package build stage')

This is a very simple script-based package (all package options explained later in this document). Repeat the same steps for the liby package with the file my-project/liby/liby containing:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

LIBY_DEPENDENCIES = ['libx']
LIBY_VERSION = '2.1.0'

And my-project/liby/libx-build containing:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

print('invoked liby package build stage')

One difference with this package is the definition of LIBY_DEPENDENCIES, which tells releng-tool to ensure that libx package is processed before liby.

With this minimal set of packages, the project’s releng-tool configuration can now be created. At the root of the project folder, create a releng configuration file with the following contents:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

packages = [
    'libx',
    'liby',
]

This sample project should be ready for a spin. While in the my-project folder, invoke releng-tool:

$ releng-tool
patching libx...
configuring libx...
building libx...
invoked libx package build stage
installing libx...
patching liby...
configuring liby...
building liby...
invoked liby package build stage
installing liby...
generating license information...
(success) completed (0:00:00)

This above output shows that the libx package’s stage are invoke followed by liby package’s stages. For the build stage in each package, each respective package script has been invoked. While this example only prints a message, more elaborate scripts can be made to handle a package’s source to build.

To clean the project, a releng-tool clean request can be invoked:

$ releng-tool clean

Now the project is in a state again to perform a fresh build. This concludes the initial getting started example. Feel free to remove the example project and prepare for steps to make a real releng-tool project.

Actually getting started¶

Start building a new releng-tool project by creating the following root and package folders for a project, and venture into the root folder:

mkdir -p <my-project>/package
cd <my-project>/

Inside the root folder, create a releng-tool configuration file releng with the following skeleton content:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

packages = [
]

This configuration defines a releng-tool project with no packages. Start to change this by adding a package or more for each a module to be built. For example, packages can be individual static libraries, simple asset fetching (e.g. image/document downloading) and more.

For each package, create a new folder inside <my-project>/package to represent the package.

mkdir <my-project>/package/<my-package>

Inside each newly created package, create a package definition file. This (Python-based) file will be named with the same name as the package folder (i.e. <my-project>/package/<my-package>/<my-package>). An initial skeleton structure for the definition file is as follows:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

MY_PACKAGE_DEPENDENCIES = []
MY_PACKAGE_LICENSE = ['<license name>']
MY_PACKAGE_LICENSE_FILES = ['<license file>']
MY_PACKAGE_SITE = '<location for sources>'
MY_PACKAGE_TYPE = '<package-type>'
MY_PACKAGE_VERSION = '<package-version>'

Initial changes to note:

For a package, any support package variable requires the package name as a prefix. The prefix should be a underscore-separated all-uppercase string value. In the above example, if the package name was my-package, the prefix will be MY_PACKAGE_.
One of the most important options is the version option (MY_PACKAGE_VERSION). This value is used to help manage downloaded asset names, build output directories and sometimes even revision values for source-fetching. An example of a good version value is ‘1.0’; however, the value can vary depending on the package being defined.
The site value (MY_PACKAGE_SITE) is used to identify where source/assets can be fetched from. A site can be a Git repository, a URL, SCP location, a site value supported by a custom fetch extension or more.
A helpful configuration option is the dependency list (MY_PACKAGE_DEPENDENCIES). If a package depends on another package being built, the name of the package should be listed in this option. This ensures that releng-tool will invoke package stages in the appropriate order.
While not required, it is recommended to provide license tracking for packages when possible. MY_PACKAGE_LICENSE and MY_PACKAGE_LICENSE_FILES are list values to define the list of licenses applicable to this package and the location of the respective license files found in the sources. A developer can exclude these options if desired. If a developer does provide these options, the end of the build process will compile a document of used licenses for the project.
The type of package (MY_PACKAGE_TYPE) can be used to take advantage of various package types supported by releng-tool. By default, packages are script-based, where Python scripts inside the packages are invoked during each stage (e.g. <my-project>/package/build would be invoked during the build phase). releng-tool also supports other package types such as autotools, CMake and more to limit the need to define custom scripts for common build steps. Developers can also use package types defined by included extensions (if any).

A detailed list of options can be found in common package configurations and advanced package configurations.

Once all packages have been prepared with desired package definitions options, the root configuration script (releng) should be updated to indicate which packages should be built. All packages can be defined in the packages list if desired. For example, if a project has packages liba, libb and programc, the following package list can be defined:

packages = [
    'liba',
    'libb',
    'programc',
]

Note that a developer does not need to explicitly add each component if dependencies are configured. Considering the same packages listed above, if programc depends on both liba and libb packages, only programc needs to be explicitly listed:

packages = [
    'programc',
]

When processing, both liba and libb packages will be implicitly loaded and processed like any other package.

Once all the project’s packages are ready, a developer can try their luck by attempting to perform a build:

releng-tool

A developer may not have luck on the first go. Tweaks may be needed on the package definitions, custom scripts (if used) and issues found in sources. A developer may invoke releng-tool multiple times to attempt to continue the build process for a project. A developer may wish to use the clean option to remove an existing extracted sources/partially built sources:

releng-tool clean

Or start from a completely fresh state using mrproper to remove any built images and file mode flags:

releng-tool mrproper

There may also be times if a single project needs to be cleaned:

releng-tool my-package-clean

Consult the user’s guide for more action information.

Eventually the project should be in a good state that each package is being built as expected. Now a user can decide on what to do with the resulting files. After the invoke of a releng-tool process, it is typical for final binaries, public includes, etc. to be found inside the <root>/output/target directory. If a developer only desires to manually take the assets from this folder and distribute/store them, no additional steps are required. However, it may be common that a developer wants to package some assets (whether it be a tar/zip/etc. container, pkg/rpm/etc. package or more). A developer could deal with such a capability outside the releng-tool process; but if a developer wishes to hook the end of the process, a post-processing script can be used.

A developer may create a post-processing file releng-post-build in the root directory. On the completion of processing each package, the post-processing script will be invoked. It is important to note that the post-processing script may be invoked multiple times if a user attempts to rebuild the project. For example, if the file releng-post-build has the following contents:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

print('project target directory: {}'.format(TARGET_DIR))

The target directory will be output to standard out at the end of the build. A developer may wish to define their own Python script to decide on how to package the contents found in TARGET_DIR (see also script helpers for helper variables/functions).

Project configuration¶

A releng-tool project defines its configuration options inside the a releng file at the root of a project. The primary configuration option for a developer to define is packages, which is used to hold a list of packages to be processed:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

packages = [
    'package-a',
    'package-b',
    'package-c',
]

A series of additional configuration options are available to be defined inside the project’s configuration. A list of common configuration options are as follows:

`default_internal`	A flag to indicate that projects are implicitly loaded as internal projects. By default, packages not explicitly configured as internal or external are assumed to be external packages. default_internal = True See also internal and external packages.
`extensions`	A list of extensions to load before processing a releng-tool project. If an extension cannot be loaded, releng-tool will stop with detailed information. extensions = [ 'ext-a', 'ext-b', ] See also loading extensions.
`external_packages`	A list of external package locations. By default, packages for a project will be searched for in root directory’s package folder (`<root>/package`). In some build environments, some packages may be required or may be preferred to be located in another location/repository. To allow packages to be loaded from another package container directory, one or more package locations can be provided. For example: external_packages = [ os.environ['MY_EXTERNAL_PKG_DIR'], ]
`license_header`	As the releng-tool build process is finalized, a license document can be generated containing each package’s license information. If a developer wishes to add a custom header to the generated document, a header can be defined by project’s configuration. For example: license_header = 'my leading content' See also licenses.
`packages`	A list of packages to process. Packages listed will be processed by releng-tool till their completion. Package dependencies not explicitly listed will be automatically loaded and processed as well. packages = [ 'package-a', 'package-b', 'package-c', ]
`prerequisites`	A list of host tools to check for before running a releng-tool project. Allows a developer to identify tools to check and fail-fast if missing, instead of waiting for a stage which requires a specific tool and failing later during a building, packaging, etc. phase. prerequisites = [ 'tool-a', 'tool-b', 'tool-c', ]
`sysroot_prefix`	Define a custom sysroot prefix to provide to packages during their configuration, build and installation stages. By default, the sysroot prefix is set to `/usr` or `\usr` (based on the system’s default character to separate pathname components). sysroot_prefix = '/usr' See also `LIBFOO_PREFIX`.
`url_mirror`	Specifies a mirror base site to be used for URL fetch requests. If this option is set, any URL fetch requests will first be tried on the configured mirror before attempting to acquired from the defined site in a package definition. url_mirror = 'ftp://mirror.example.org/data/'

A list of more advanced configuration options are as follows:

`cache_ext`	A transform for cache extension interpreting. This is an advanced configuration and is not recommended for use except for special use cases outlined below. When releng-tool fetches assets from remote sites, the site value can used to determine the resulting filename of a cached asset. For example, downloading an asset from `https://example.org/my-file.tgz`, the locally downloaded file will result in a `.tgz` extension; however, not all defined sites will result in an easily interpreted cache extension. While releng-tool will attempt its best to determine an appropriate extension value to use, some use cases may not be able to be handled. To deal with these cases, a developer ca define a transform method to help translate a site value into a known cache extension value. Consider the following example: a host is used to acquire assets from a content server. The URI to download an asset uses a unique request format `https://static.example.org/fetch/25134`. releng-tool may not be able to find the extension for the fetched asset, but if a developer knows the expected archive types for these calls, a custom transform can be defined. For example: def my_translator(site): if 'static.example.org' in site: return 'tgz' return None cache_ext = my_translator The above transform indicates that all packages using the `static.example.org` site will be `tgz` archives.
`extra_license_exceptions`	A dictionary to define extra license exceptions that are permitted in package definitions. Packages which define license exceptions in a `LIBFOO_LICENSE` option are expected to use SPDX License Exceptions. If not, a warning is generated by default. A project can define their own custom exceptions by adding them into a project’s `extra_license_exceptions` option to avoid this warning: extra_license_exceptions = { 'My-Exception-ID': 'Exception Name', }
`extra_licenses`	A dictionary to define extra licenses that are permitted in package definitions. Packages which define licenses in a `LIBFOO_LICENSE` option are expected to use a licensed defined in the SPDX License List. If not, a warning is generated by default. A project can define their own custom license by adding them into a project’s `extra_licenses` option to avoid this warning: extra_licenses = { 'My-License-ID': 'License Name', }
`override_extract_tools`	A dictionary to be provided to map an extension type to an external tool to indicate which tool should be used for extraction. For example, when a `.zip` archive is being processed for extraction, releng-tool will internally extract the archive; however, a user may wish to override this tool with their own extraction utility. Consider the following example: override_extract_tools = { 'zip': '/opt/my-custom-unzip {file} {dir}', } Note that the use of an override option should only be used in special cases (see also configuration overrides).
`override_revisions`	Allows a dictionary to be provided to map a package name to a new revision value. Consider the following example: a project defines `module-a` and `module-b` packages with package `module-b` depending on package `module-a`. A developer may be attempting to tweak package `module-b` on the fly to test a new capabilities against the current stable version of `module-a`; however, the developer does not want to explicitly change the revision inside package `module-b`’s definition. To avoid this, an override can be used instead: override_revisions = { 'module-b': '<test-branch>', } The above example shows that package `module-b` will fetch using a test branch instead of what is defined in the actual package definition. Note that the use of an override option should only be used in special cases (see also configuration overrides).
`override_sites`	A dictionary to be provided to map a package name to a new site value. There may be times where a host may not have access to a specific package site. To have a host to use a mirror location without having to adjust the package definition, the site override option can be used. For example, consider a package pulls from site `git@example.com:myproject.git`; however, the host `example.com` cannot be access from the host machine. If a mirror location has been setup at `git@example.org:myproject.git`, the following override can be used: override_sites = { '<pkg>': 'git@example.org:mywork.git', } Note that the use of an override option should only be used in special cases (see also configuration overrides).
`quirks`	A list of configuration quirks to apply to deal with corner case scenarios which can prevent releng-tool operating on a host system. quirks = [ 'releng.<special-quirk-id>', ] For a list of available quirks, see quirks.
`urlopen_context`	Allows a project to specify a custom SSL context [10] to apply for URL fetch requests. This can be useful for environments which may experience `CERTIFICATE_VERIFY_FAILED` errors when attempting to fetch files. A custom SSL context can be created and tailored for a build environment. For example: import ssl ... urlopen_context = ssl.create_default_context()

Environment variables¶

Tip

Avoid using external environment variables for a project to configure package options such as compiler flags or interpreters. Managing these options inside a releng-tool project configuration or package definitions can improve configuration management.

When configuration, package definitions or various scripts are invoked by releng-tool, the following environment variables are available:

`BUILD_DIR`	The build directory.
`CACHE_DIR`	The cache directory.
`DL_DIR`	The download directory.
`HOST_BIN_DIR`	The host directory’s prefixed bin directory.
`HOST_DIR`	The host directory.
`HOST_INCLUDE_DIR`	The host directory’s prefixed include directory.
`HOST_LIB_DIR`	The host directory’s prefixed library directory.
`IMAGES_DIR`	The images directory.
`LICENSE_DIR`	The licenses directory. See also licenses.
`NJOBS`	Number of calculated jobs to allow at a given time. Unless explicitly set by a system builder on the command line, the calculated number of jobs should be equal to the number of physical cores on the host. When building a specific package and the package overrides the number of jobs to use, the package-defined count will be used instead. This configuration will always be a value of at least one (`1`).
`NJOBSCONF`	Number of jobs to allow at a given time. Unlike `NJOBS`, `NJOBSCONF` provides the requested configured number of jobs to use. The value may be set to zero (`0`) to indicate an automatic detection of jobs to use. This can be useful for tools which have their own automatic job count implementation and do not want to rely on the value defined by `NJOBS`. When building a specific package and the package overrides the number of jobs to use, the package-defined count will be used instead.
`OUTPUT_DIR`	The output directory.
`PKG_BUILD_BASE_DIR`	The directory for a specific package’s base directory for buildable content. In most cases, this value will be the same as `PKG_BUILD_DIR`; however, if `LIBFOO_BUILD_SUBDIR` is configured, `PKG_BUILD_DIR` will include the configured sub-directory, where this base directory will not. See also `PKG_BUILD_DIR`.
`PKG_BUILD_DIR`	The directory for a specific package’s buildable content. See also `PKG_BUILD_BASE_DIR` and `PKG_BUILD_OUTPUT_DIR`.
`PKG_BUILD_OUTPUT_DIR`	The directory for where a package’s build output will be stored. See also `PKG_BUILD_DIR`.
`PKG_CACHE_DIR`	The location of the cache directory for a package. If a package defines a fetch from a repository which can be locally cached, this cache directory represents the location where the local cache of content will be held. For example, if a provide defines a Git-based site, a local cache of the Git repository will be stored in this location. Typically, packages should not need to operate on the cache directory except for advanced cases.
`PKG_CACHE_FILE`	The location of the cache file for a package. If a package defines a fetch of an archive from a remote source, after the fetch stage is completed, the archive can be found in this location. For example, if a package defines a site `https://www.example.com/test.tgz`, the resulting cache file may be `<root>/output/dl/test-1.0.tgz`.
`PKG_DEFDIR`	The package’s definition directory. For example, for a package `test`. the definition directory would be `<root>/package/test`.
`PKG_DEVMODE`	Whether the package is configured for development mode. If a package is configured for development mode, the environment variable will be set to a value of one (i.e. `PKG_DEVMODE=1`). See also development mode.
`PKG_INTERNAL`	Whether or not the package is considered “internal”. If internal, the environment variable will be set to a value of one (i.e. `PKG_INTERNAL=1`). See also internal and external packages.
`PKG_LOCALSRCS`	Whether the package is configured for local-sources mode. If a package is configured for local-sources, the environment variable will be set to a value of one (i.e. `PKG_LOCALSRCS=1`). See also local-sources mode.
`PKG_NAME`	The name of the package.
`PKG_REVISION`	The site revision of the package. See also `LIBFOO_REVISION`.
`PKG_SITE`	The site of the package. See also `LIBFOO_SITE`.
`PKG_VERSION`	The version of the package. See also `LIBFOO_VERSION`.
`PREFIX`	The sysroot prefix for the package.
`PREFIXED_HOST_DIR`	The host directory with the prefix applied.
`PREFIXED_STAGING_DIR`	The staging area directory with the prefix applied.
`PREFIXED_TARGET_DIR`	The target area directory with the prefix applied.
`RELENG_CLEAN`	Flag set if performing a clean request.
`RELENG_DEBUG`	Flag set if debug-related information should be shown.
`RELENG_DEVMODE`	The development mode or flag set if in development mode.
`RELENG_DISTCLEAN`	Flag set if performing an extreme pristine clean request.
`RELENG_FORCE`	Flag set if performing a forced request from the command line.
`RELENG_LOCALSRCS`	Flag set if in local-sources mode.
`RELENG_MRPROPER`	Flag set if performing a pristine clean request.
`RELENG_REBUILD`	Flag set if performing a re-build request.
`RELENG_RECONFIGURE`	Flag set if performing a re-configuration request.
`RELENG_REINSTALL`	Flag set if performing a re-install request.
`RELENG_TARGET_PKG`	The name of the target package (if any) provided by the command line.
`RELENG_VERBOSE`	Flag set if verbose-related information should be shown.
`RELENG_VERSION`	The version of releng-tool.
`ROOT_DIR`	The root directory.
`STAGING_BIN_DIR`	The staging area directory’s prefixed bin directory.
`STAGING_DIR`	The staging area directory.
`STAGING_INCLUDE_DIR`	The staging area directory’s prefixed include directory.
`STAGING_LIB_DIR`	The staging area directory’s prefixed library directory.
`SYMBOLS_DIR`	The symbols area directory.
`TARGET_BIN_DIR`	The target area directory’s prefixed bin directory.
`TARGET_DIR`	The target area directory.
`TARGET_INCLUDE_DIR`	The target area directory’s prefixed include directory.
`TARGET_LIB_DIR`	The target area directory’s prefixed library directory.

Package-specific environment variables are also available if another package or script needs to rely on the (generated) configuration of another package. The following package-specific environment variables are available for use (where <PKG> translates to a releng-tool’s determined package key):

`<PKG>_BUILD_DIR`	The directory for a defined package’s buildable content.
`<PKG>_BUILD_OUTPUT_DIR`	The directory for where a defined package’s build output will be stored.
`<PKG>_DEFDIR`	The directory where a defined package’s definition is stored.
`<PKG>_NAME`	The name of the package.
`<PKG>_REVISION`	The revision of a defined package.
`<PKG>_VERSION`	The version of a defined package.

releng-tool also accepts environment variables for configuring specific features of the releng-tool process. The following environment variables are accepted:

`RELENG_ASSETS_DIR=<dir>`	The asset directory to use.
`RELENG_CACHE_DIR=<dir>`	The cache directory to use.
`RELENG_DL_DIR=<dir>`	The download directory to use.
`RELENG_IGNORE_RUNNING_AS_ROOT=1`	Suppress the warning generated when running releng-tool with a root user.

Packages¶

Packages are defined inside the package/ directory. There is no explicit limit on the total number of packages a project can have. Packages can consist of libraries, programs or even basic assets. Package names are recommended to be lower-case with dash-separated (-) separators (if needed). For example, package-a is recommended over PackageA or package_a; however, the choice is up to the developer making the releng-tool project.

When making a package, a container folder for the package as well as a package definition file needs to be made. For example, if the package is package-a, the file package/package-a/package-a should exist. Package definition files are Python-based, thus the following leading header is recommended:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

Inside the definition file, a series of configuration options can be set to tell releng-tool how to work with the defined package. Each option is prefixed with a variable-safe variant of the package name. The prefix value will be an uppercase string based on the package name with special characters converted to underscores. For example, package-a will have a prefix PACKAGE_A_. For a package to take advantage of a configuration option, the package definition will add a variable entry with the package’s prefix followed by the supported option name. Considering the same package with the name package-a (and prefix PACKAGE_A_), to use the LIBFOO_VERSION configuration option, the following can be defined (PACKAGE_A_VERSION):

#!/usr/bin/env python
# -*- coding: utf-8 -*-

PACKAGE_A_VERSION = '1.0.0'

More details on available configuration options are as follows.

Common package configurations¶

The following outlines common configuration options available for packages.

LIBFOO_DEPENDENCIES¶

List of package dependencies a given project has. If a project depends on another package, the package name should be listed in this option. This ensures releng-tool will process packages in the proper order. The following shows an example package libc being dependent on liba and libb being processed first:

LIBC_DEPENDENCIES = ['liba', 'libb']

LIBFOO_INSTALL_TYPE¶

Defines the installation type of this package. A package may be designed to be built and installed for just the target area, the stage area, both or maybe in the host directory. The following options are available for the installation type:

Type	Description
`host`	The host directory.
`images`	The images directory.
`staging`	The staging area.
`staging_and_target`	Both the staging an target area.
`target`	The target area.

The default installation type is target.

LIBFOO_INSTALL_TYPE = 'target'

LIBFOO_LICENSE¶

A string or list of strings outlining the license information for a package. Outlining the license of a package is always recommended (but not required). It is recommended to use SPDX registered licenses.

LIBFOO_LICENSE = ['GPL-2.0-only', 'MIT']

LIBFOO_LICENSE = 'GPL-2.0-or-later WITH Bison-exception-2.2'

LIBFOO_LICENSE = 'Proprietary'

LIBFOO_LICENSE_FILES¶

A string or list of strings identifying the license files found inside the package sources which match up to the defined LICENSE entries (respectively). Listing the license(s) of a package is always recommended (but not required).

LIBFOO_LICENSE_FILES = [
    'LICENSE.GPLv2',
    'LICENSE.MIT',
]

LIBFOO_LICENSE_FILES = 'LICENSE'

LIBFOO_SITE¶

The site where package sources/assets can be found. The site can be a URL of an archive, or describe a source control URL such as Git or SVN. The following outline a series of supported site definitions:

Type	Prefix/Postfix
Bazaar	`bzr+`
CVS	`cvs+`
Git	`git+` or `.git`
Mercurial	`hg+`
rsync	`rsync+`
SCP	`scp+`
SVN	`svn+`
URL	(wildcard)

Examples include:

LIBFOO_SITE = 'https://example.com/libfoo.git'
LIBFOO_SITE = 'cvs+:pserver:anonymous@cvs.example.com:/var/lib/cvsroot mymodule'
LIBFOO_SITE = 'svn+https://svn.example.com/repos/libfoo/c/branches/libfoo-1.2'
LIBFOO_SITE = 'https://www.example.com/files/libfoo.tar.gz'
LIBFOO_SITE = {
    DEFAULT_SITE: 'https://pkgs.example.com/releases/libfoo-${LIBFOO_VERSION}.tar.gz',
    '<mode>': 'https://git.example.com/libfoo.git',
}

A developer can also use LIBFOO_VCS_TYPE to explicitly define the version control system type without the need for a prefix/postfix entry. The use of a dictionary value is only useful when operating in development mode.

For more information on each type’s formatting, see site definitions.

Using a specific type will create a dependency for a project that the respective host tool is installed on the host system. For example, if a Git site is set, the host system will need to have git installed on the system.

If no site is defined for a package, it will be considered a virtual package (i.e. has no content). If applicable, loaded extensions may provide support for custom site protocols.

Specifying a local site value with local will automatically configure a VCS-type of local (see LIBFOO_VCS_TYPE for more information).

LIBFOO_TYPE¶

The package type. The default package type is a (Python) script-based package; however, releng-tool also provides a series of helper package types for common frameworks. The following outline a series of supported type definitions:

Type	Value
Autotools	`autotools`
CMake	`cmake`
Make	`make`
Python	`python`
SCons	`scons`
Script	`script`

For example:

LIBFOO_TYPE = 'script'

If no type is defined for a package, it will be considered a script-based package. If applicable, loaded extensions may provide support for custom site protocols.

Using a specific type will create a dependency for a project that the respective host tool is installed on the host system. For example, if a CMake type is set, the host system will need to have cmake installed on the system.

LIBFOO_VERSION¶

The version of the package. Typically the version value should be formatted in a semantic versioning style; however, it is up to the developer to decide the best version value to use for a package. It is important to note that the version value is used to determine build output folder names, cache files and more.

LIBFOO_VERSION = '1.0.0'

For some VCS types, the version value will be used to acquire a specific revision of sources. If for some case the desired version value cannot be gracefully defined (e.g. libfoo-v1.0 will produce output directories such as libfoo-libfoo-v1.0), LIBFOO_REVISION can be used.

Advanced package configurations¶

The following outlines more advanced configuration options available for packages.

LIBFOO_BUILD_SUBDIR¶

Sub-directory where a package’s extracted sources holds its buildable content. Sources for a package may be nested inside one or more directories. A package can specify the sub-directory where the configuration, build and installation processes are invoked from.

LIBFOO_BUILD_SUBDIR = 'subdir'

LIBFOO_DEVMODE_IGNORE_CACHE¶

Flag value to explicitly indicate that a package should ignore any generated cache file when operating in development mode. In most cases, users want to take advantage of cached sources to prevent having to re-fetch the same content again between builds. However, some packages may be configured in a way where their request for a package’s contents varies from a fresh stage. For example, when pulling from a branch, releng-tool will not attempt to re-fetch from a site since a cached content has already been fetched. If a developer configures a package to use a revision value with dynamic content, they may wish to use this option to have a user always force fetching new content from a clean state.

LIBFOO_DEVMODE_IGNORE_CACHE = True

By default, this option is not defined and results can vary based off the site type being fetched. In most cases, fetch operations will treat the default case of this option as disabled (False). DVCS site types may elect to enable this option by default (True) if the target revision is a branch.

LIBFOO_DEVMODE_REVISION¶

Specifies a development revision for a package. When a project is being built in development mode, the development revision is used over the configured LIBFOO_REVISION value. If a development revision is not defined for a project, a package will still use the configured LIBFOO_REVISION while in development mode.

LIBFOO_DEVMODE_REVISION = 'feature/alpha'

See also LIBFOO_REVISION and LIBFOO_VERSION.

LIBFOO_EXTENSION¶

Specifies a filename extension for the package. A package may be cached inside the download directory to be used when the extraction phase is invoked. releng-tool attempts to determine the most ideal extension for this cache file; however some cases the detected extension may be incorrect. To deal with this situation, a developer can explicitly specify the extension value using this option.

LIBFOO_EXTENSION = 'tgz'

LIBFOO_EXTERNAL¶

Flag value to explicitly indicate that a package is an external package. External packages will generate warnings if hashes, an ASCII-armor or licenses are missing. By default, packages are considered external unless explicitly configured to be internal.

LIBFOO_EXTERNAL = True

LIBFOO_EXTOPT¶

Specifies extension-specific options. Packages wishing to take advantage of extension-specific capabilities can forward options to extensions by defining a dictionary of values.

LIBFOO_EXTOPT = {
    'option-a': True,
    'option-b': 'value',
}

LIBFOO_EXTRACT_TYPE¶

Specifies a custom extraction type for a package. If a configured extension supports a custom extraction capability, the registered extraction type can be explicitly registered in this option.

LIBFOO_EXTRACT_TYPE = 'ext-custom-extract'

LIBFOO_FETCH_OPTS¶

Provides a means to pass command line options into the fetch process. This option can be defined as a dictionary of string pairs or a list with strings – either way defined will generate argument values which may be included in a fetch event. This field is optional. Not all site types may support this option.

LIBFOO_FETCH_OPTS = {
    # adds "--option value" to the command
    '--option': 'value',
}

# (or)

LIBFOO_FETCH_OPTS = [
    # adds "--some-option" to the command
    '--some-option',
]

LIBFOO_FIXED_JOBS¶

Explicitly configure the total number of jobs a package can use. The primary use case for this option is to help limit the total number of jobs for a package that cannot support a large or any parallel build environment.

LIBFOO_FIXED_JOBS = 1

LIBFOO_GIT_CONFIG¶

Apply additional repository-specific Git configuration settings (git config) after a Git repository cache has been initialized. By default, no repository-specific configurations are introduced (i.e. all Git calls will use the global configuration set).

LIBFOO_GIT_CONFIG = {
   'core.example': 'value',
}

LIBFOO_GIT_DEPTH¶

Limit fetching for a Git-based source to the specified number of commits. The value provided will be used with the --depth argument. By default, the depth will be set to a value of 1. If a developer wishes use fetch all commits from all refspecs, a developer can specify a value of 0.

While the default depth is a value of 1, an exception is made when the depth is not explicitly set and the LIBFOO_REVISION value defined is a hash. For this case, if the revision is not found with the implicitly-defined shallow depth of 1, the entire history of the repository will be fetched.

LIBFOO_GIT_DEPTH = 0

LIBFOO_GIT_REFSPECS¶

List of addition refspecs to fetch when using a git VCS type. By default, a Git fetch request will acquire all heads and tags refspecs. If a developer wishes use revisions from different refspecs (for example, a pull request), a developer can specify the additional refspecs to acquire when fetching.

LIBFOO_GIT_REFSPECS = ['pull/*']

LIBFOO_GIT_SUBMODULES¶

Flag value to indicate whether or not a package’s Git submodules should be fetched/extracted during a package’s own fetch/extraction stages. By default, submodules are not fetched. Ideally, any dependencies for a package are recommended to be defined in their own individual package; however, this may not be ideal for all environments. When configured, submodules will be cached in the same fashion as other Git-based packages. Note that submodule caching is specific to the repository being processed (i.e. they cannot be “shared” between other packages). If multiple packages have the same dependency defined through a submodule, it is recommended to create a new package and reference its contents instead.

LIBFOO_GIT_SUBMODULES = True

LIBFOO_GIT_VERIFY_REVISION¶

Flag value to indicate whether or not the target revision is required to be signed before it can be used. When this value is set, the configured revision for a repository will not be extracted unless the GPG signature. This includes if the public key for the author is not registered in the local system or if the target revision is not signed.

LIBFOO_GIT_VERIFY_REVISION = True

LIBFOO_HOST_PROVIDES¶

Hints at what host tools this package may be providing. A project may have a series of prerequisites, which are checked at the start of a run. This is to help ensure required host tools are available before attempting to build a project. If a package is designed to provide a host package (e.g. when using LIBFOO_INSTALL_TYPE with the host option), these packages can provide tools other packages may rely on. However, prerequisites checks will occur before these packages may be built, preventing a build from running. This option allows a developer to hint at what tools a host package may provide. By specifying the name of a tool in this option, an initial prerequisites check will not fail if a tool is not available at the start of a run.

LIBFOO_HOST_PROVIDES = 'some-tool'

# (or)

LIBFOO_HOST_PROVIDES = [
    'tool-a',
    'tool-b',
    'tool-c',
]

LIBFOO_INTERNAL¶

Flag value to explicitly indicate that a package is an internal package. Internal packages will not generate warnings if hashes, an ASCII-armor or licenses are missing. When configured in local-sources mode, package sources are searched for in the local directory opposed to site fetched sources. By default, packages are considered external unless explicitly configured to be internal.

LIBFOO_INTERNAL = True

LIBFOO_NO_EXTRACTION¶

Warning

If LIBFOO_NO_EXTRACTION is configured for a package, the package cannot define additional hashes, configure an ASCII-armor, define a list of LIBFOO_LICENSE_FILES to manage or expect to support various actions (such as building; since no sources are available).

Flag value to explicitly indicate that a package should not extract the package contents. This feature is primarily used when using releng-tool to fetch content for one or more packages (into DL_DIR) to be used by another package the releng-tool project defines.

LIBFOO_NO_EXTRACTION = True

Limitations exist when using the LIBFOO_NO_EXTRACTION option. Since releng-tool will not be used to extract a package’s archive (if any), hash entries for files found inside the archive cannot be checked against. If any files other than the archive itself is listed, releng-tool will stop processing due to a hash check failure. In addition, since releng-tool does not have the extracted contents of an archive, it is unable to acquire a copy of the project’s license file. Specifying LIBFOO_LICENSE_FILES for projects with the no-extraction flag enabled will result in a warning. By default, this option is disabled with a value of False.

LIBFOO_PREFIX¶

Specifies the sysroot prefix value to use for the package. An explicitly provided prefix value will override the project-defined or default sysroot prefix value.

LIBFOO_PREFIX = '/usr'

LIBFOO_REVISION¶

Specifies a revision value for a package. When a package fetches content using source management tools, the revision value is used to determine which sources should be acquired (e.g. a tag). If a revision is not defined package, a package will use the configured LIBFOO_VERSION.

LIBFOO_REVISION = 'libfoo-v2.1'

For users planning to take advantage of development mode capabilities, multiple revisions can be configured based off the mode:

LIBFOO_REVISION = {
    DEFAULT_REVISION: 'libfoo-v2.1',
    'develop': 'main',
}

LIBFOO_SKIP_REMOTE_CONFIG¶

Flag value to explicitly indicate that a package should not attempt to load any package configurations which may be defined in the package’s source. A package, by default, has the ability to load configuration information from a package’s source. If the package includes a .releng-tool file at the root of their sources, supported configuration options that have not been populated will be registered into the package before invoking a package’s configuration stage.

LIBFOO_SKIP_REMOTE_CONFIG = True

LIBFOO_SKIP_REMOTE_SCRIPTS¶

Flag value to explicitly indicate that a package should not attempt to load any package scripts which may be defined in the package’s source. Typically, a script-based package will load configuration, build, etc. scripts from its package definition folder. If a script-based package is missing a stage script to invoke and finds an associated script in the package’s source, the detected script will be invoked. For example, if libfoo package may attempt to load a libfoo-configure script for a configuration stage. In the event that the script cannot be found and remote scripting is permitted for a package, the script (if exists) releng-configure will be loaded from the root of the package’s contents.

LIBFOO_SKIP_REMOTE_CONFIG = True

LIBFOO_STRIP_COUNT¶

Specifies the strip count to use when attempting to extract sources from an archive. By default, the extraction process will strip a single directory from an archive (value: 1). If a package’s archive has no container directory, a strip count of zero can be set; likewise if an archive contains multiple container directories, a higher strip count can be set.

LIBFOO_STRIP_COUNT = 1

LIBFOO_VCS_TYPE¶

Explicitly sets the version control system type to use when acquiring sources. releng-tool attempts to automatically determine the VCS type of a package based off a LIBFOO_SITE value. In some scenarios, a site value may be unable to specify a desired prefix/postfix. A developer can instead explicitly set the VCS type to be used no matter what the site value is configured as.

Supported types are as follows:

bzr (Bazaar)
cvs (CVS)
git (Git)
hg (Mercurial)
local (no VCS; local interim-development package)
none (no VCS; virtual package)
rsync (rsync)
scp (SCP)
svn (SVN)
url (URL)

LIBFOO_VCS_TYPE = 'git'

If a project registers a custom extension which provides a custom VCS type, the extension type can be set in this option.

For users planning to take advantage of development mode capabilities with mode-specific sites, users can provide an explicit VCS type based off a configured mode:

LIBFOO_VCS_TYPE = {
    DEFAULT_REVISION: 'git',
    'legacy': 'cvs',
}

Using a specific type will create a dependency for a project that the respective host tool is installed on the host system. For example, if a Git VCS-type is set, the host system will need to have git installed on the system.

Note that the local type is designed to be a special/development-helper type only. When set, this option allows placing the sources of a package directly inside a local folder inside the definition folder. This approach is similar to using local-sources mode where it avoids the need to have the module content located in a site to be fetched – specifically, for initial development/testing/training scenarios. It is never recommended to store the package’s “main content” inside a releng-tool project, thus using the local type will always generate a warning message.

Package bootstrapping and post-processing¶

Every package, no matter which package LIBFOO_TYPE is defined, can create a bootstrapping or post-processing script to invoke before a package starts a configuration stage or after a package has completed an installation stage, respectively.

The existence of a <package>-bootstrap inside a package directory will trigger the bootstrapping stage for the package. An example bootstrapping script (libfoo-bootstrap) can be as follows:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

print('perform bootstrapping work')

The existence of a <package>-post inside a package directory will trigger the post-processing stage for the package. An example post-processing script (libfoo-post) can be as follows:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

print('perform post-processing work')

Bootstrapping or post-processing scripts for a package are optional; thus, if no scripts are provided for a package, no bootstrapping or post-processing will be performed for the package.

See also script helpers for helper functions/variables available for use.

Site definitions¶

The following outlines the details for defining supported site definitions. If attempting to use an extension-provided site type, please consult the documentation provided by said extension.

Note

All site values can be defined with a prefix value (e.g. git+ for Git sources) or postfix value; however, this is optional if a package wishes to use the LIBFOO_VCS_TYPE option.

Bazaar Site¶

To define a Bazaar-based location, the site value must be prefixed with a bzr+ value. A site can be defined as follows:

LIBFOO_SITE = 'bzr+ssh://example.com/project/trunk'
# (or)
LIBFOO_SITE = 'bzr+lp:<project>'

The value after the prefix is a path which will be provided to a bzr export call [1]. Content from a Bazaar repository will be fetched and archived into a file during fetch stage. Once a cached archive is made, the fetch stage will be skipped unless the archive is manually removed.

CVS Site¶

To define a CVS-based location, the site value must be prefixed with a cvs+ or other common CVSROOT value. A site can be defined as follows:

LIBFOO_SITE = ':pserver:anonymous@cvs.example.com:/var/lib/cvsroot mymodule'
# (or)
LIBFOO_SITE = 'cvs+:ext:cvs@cvs.example.org:/usr/local/cvsroot mymodule'

The value after the prefix is a space-separated pair, where the first part represents the CVSROOT [3] to use and the second part specifies the CVS module [2] to use. Content from a CVS repository will be fetched and archived into a file during fetch stage. Once a cached archive is made, the fetch stage will be skipped unless the archive is manually removed.

Git Site¶

To define a Git-based location, the site value must be prefixed with a git+ value or postfixed with the .git value. A site can be defined as follows:

LIBFOO_SITE = 'https://example.com/libfoo.git'
# (or)
LIBFOO_SITE = 'git+git@example.com:base/libfoo.git'

The site value (less prefix, if used) is used as a Git remote [4] for a locally managed cache source. Git sources will be cached inside the cache directory on first-run. Future runs to fetch a project’s source will use the cached Git file system. If a desired revision exists, content will be acquired from the cache location. If a desired revision does not exist, the origin remote will be fetched for the new revision (if it exists).

Local Site¶

To define a package to use local site/sources, the site value can be set to local. A local site can be defined as follows:

LIBFOO_SITE = 'local'

This is equivalent to configuring LIBFOO_VCS_TYPE to a local VCS type as well. Note that a local package is intended for development/testing/training purposes. See LIBFOO_VCS_TYPE for more information.

Mercurial Site¶

To define a Mercurial-based location, the site value must be prefixed with a hg+ value. A site can be defined as follows:

LIBFOO_SITE = 'hg+https://example.com/project'

The value after the prefix is used as the SOURCE in an hg clone call [6]. Mercurial sources will be cached inside the cache directory on first-run. Future runs to fetch a project’s source will use the cached Mercurial repository. If a desired revision exists, content will be acquired from the cache location. If a desired revision does not exist, the origin remote will be pulled for the new revision (if it exists).

rsync Site¶

To define an rsync-based location, the site value must be prefixed with an rsync+ value. A site can be defined as follows:

LIBFOO_SITE = 'rsync+<source>'

The value of <source> will be provided to a rsync call’s [7] SRC value. Fetched content will be stored in an archive inside the dl directory. Once fetched, the fetch stage will be skipped unless the archive is manually removed. By default, the --recursive argument is applied. Adding or replacing options can be done by using the LIBFOO_FETCH_OPTS option.

SCP Site¶

To define an SCP-based location, the site value must be prefixed with a scp+ value. A site can be defined as follows:

LIBFOO_SITE = 'scp+[user@]host:]file'

The value after the prefix is a path which will be provided to a scp call’s [8] source host value. The SCP site only supports copying a file from a remote host. The fetched file will be stored inside the dl directory. Once fetched, the fetch stage will be skipped unless the file is manually removed.

SVN Site¶

To define a Subversion-based location, the site value must be prefixed with a svn+ value. A site can be defined as follows:

LIBFOO_SITE = 'svn+https://svn.example.com/repos/libfoo/c/branches/libfoo-1.2'

The value after the prefix is a path which will be provided to a svn checkout call [9]. Content from a Subversion repository will be fetched and archived into a file during fetch stage. Once a cached archive is made, the fetch stage will be skipped unless the archive is manually removed.

URL Site (default)¶

All packages that do not define a helper prefix/postfix value (as seen in other site definitions) or do not explicitly set a LIBFOO_VCS_TYPE value (other than url), will be considered a URL site. A URL site can be defined as follows:

LIBFOO_SITE = 'https://example.com/my-file'

The site value provided will be directly used in a URL request. URL values supported are defined by the Python’s urlopen implementation [10], which includes (but not limited to) http(s)://, ftp://, file:// and more.

Hash file¶

Note

An alternative to using a hash to validate a package’s cache is to use an ASCII-armor instead. Although users can benefit from using both validation methods, if desired.

When downloading assets from a remote instance, a package’s hash file can be used to help verify the integrity of any fetched content. For example, if a package lists a site with a my-archive.tgz to download, the fetch process will download the archive and verify its hash to a listed entry before continuing. If a hash does not match, the build process stops indicating an unexpected asset was downloaded.

It is recommended that:

Any URL-based site asset have a hash entry defined for the asset (to ensure the package sources are not corrupted or have been unexpectedly replaced).
A hash entry should exist for license files (additional sanity check if a package’s license has change).

To create a hash file for a package, add a <my-package>.hash file inside the package’s directory. The hash file should be a UTF-8 encoded file and can contain multiple hash entries. A hash entry is a 3-tuple defining the type of hash algorithm used, the hash value expected and the asset associated with the hash. A tuple entry is defined on a single line with each entry separated by whitespace characters. For example:

# my hashes
sha1 f606cb022b86086407ad735bf4ec83478dc0a2c5 my-archive.tgz
sha1 602effb4893c7504ffee8a8efcd265d86cd21609 LICENSE

Comments are permitted in the file. Lines leading with a # character or inlined leading # character after a whitespace character will be ignored.

Supported hash types will vary on the Python interpreter [5] used. Typically, this include FIPS secure hash algorithms (e.g. sha1, sha224, sha256, sha384 and sha512) as well as (but not recommended) RSA’S MD5 algorithm. For hash algorithms requiring a key length, a user can define a hash entry using the format <hash-type>:<key-length>. For example, shake_128:32. Other algorithms may be used if provided by the system’s OpenSSL library.

Multiple hash entries can be provided for the same file if desired. This is to assist in scenarios where a checked out asset’s content changes based on the system it is checked out on. For example, a text file checked out from Git may use Windows line-ending on Windows system, and Unix-line endings on other systems:

sha1 602effb4893c7504ffee8a8efcd265d86cd21609 LICENSE
sha1 9e79b84ef32e911f8056d80a311cf281b2121469 LICENSE

ASCII Armor¶

Note

An alternative to using an ASCII-armor to validate a package’s cache is to use hashes instead.

When downloading assets from a remote instance, an ASCII-armor file can be used to help verify the integrity of any fetched content. For example, if a package lists a site with a my-archive.tgz to download, the fetch process will download the archive and verify its contents with an associated ASCII-armor file (if one is provided). If the integrity of the file cannot be verified, the build process stops indicating an unexpected asset was downloaded.

To include an ASCII-armor file for a package, add a <my-package>.asc file inside the package’s directory. Verification is performed using the host system’s gpg. For verification’s to succeed, the system must already have the required public keys registered.

Script Package (default)¶

A script-based package is the most basic package type available. By default, packages are considered to be script packages unless explicitly configured to be another package type (LIBFOO_TYPE). If a developer wishes to explicitly configure a project as script-based, the following configuration can be used:

LIBFOO_TYPE = 'script'

A script package has the ability to define three Python stage scripts:

<package>-configure - script to invoke during the configuration stage
<package>-build - script to invoke during the build stage
<package>-install - script to invoke during the installation stage

An example build script (libfoo-build) can be as follows:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

releng_execute(['make'])

When a package performs a configuration, build or installation stage; the respective script (mentioned above) will be invoked. Package scripts are optional; thus, if a script is not provided for a stage, the stage will be skipped.

See also script helpers for helper functions/variables available for use an bootstrapping/post-processing for more stage script support.

Autotools Package¶

An autotools package provides support for processing a GNU Build System supported module.

LIBFOO_TYPE = 'autotools'

When an autotools package performs a configuration stage, the package may invoke autoreconf (if configured to do so) and then invoke configure. When the build stage is reached, make will be invoked followed by make install during the installation stage.

The following sections outline configuration options are available for an autotools package.

LIBFOO_AUTOTOOLS_AUTORECONF¶

Specifies whether or not the package needs to perform an autotools re-configuration. This is to assist in the rebuilding of GNU Build System files which may be broken or a patch has introduced new build script changes that need to be applied. This field is optional. By default, autoreconf is not invoked.

LIBFOO_AUTOTOOLS_AUTORECONF = True

LIBFOO_BUILD_DEFS¶

Provides a means to pass definitions into the build process. This option can is defined as a dictionary of string pairs. This field is optional.

LIBFOO_BUILD_DEFS = {
    # adds "--option=value" to the command
    '--option': 'value',
}

LIBFOO_BUILD_ENV¶

Provides a means to pass environment variables into the build process. This option is defined as a dictionary with key-value pairs where the key is the environment name and the value is the environment variable’s value. This field is optional.

LIBFOO_BUILD_ENV = {
    'OPTION': 'VALUE',
}

LIBFOO_BUILD_OPTS¶

Provides a means to pass command line options into the build process. This option can be defined as a dictionary of string pairs or a list with strings – either way defined will generate argument values to include in the build event. This field is optional.

LIBFOO_BUILD_OPTS = {
    # adds "--option value" to the command
    '--option': 'value',
}

# (or)

LIBFOO_BUILD_OPTS = [
    # adds "--some-option" to the command
    '--some-option',
]

LIBFOO_CONF_DEFS¶

Provides a means to pass definitions into the configuration process. This option can is defined as a dictionary of string pairs. This field is optional.

LIBFOO_CONF_DEFS = {
    # adds "--option=value" to the command
    '--option': 'value',
}

LIBFOO_CONF_ENV¶

Provides a means to pass environment variables into the configuration process. This option is defined as a dictionary with key-value pairs where the key is the environment name and the value is the environment variable’s value. This field is optional.

LIBFOO_CONF_ENV = {
    'OPTION': 'VALUE',
}

LIBFOO_CONF_OPTS¶

Provides a means to pass command line options into the configuration process. This option can be defined as a dictionary of string pairs or a list with strings – either way defined will generate argument values to include in the configuration event. This field is optional.

LIBFOO_CONF_OPTS = {
    # adds "--option value" to the command
    '--option': 'value',
}

# (or)

LIBFOO_CONF_OPTS = [
    # adds "--some-option" to the command
    '--some-option',
]

LIBFOO_INSTALL_DEFS¶

Provides a means to pass definitions into the installation process. This option can is defined as a dictionary of string pairs. This field is optional.

LIBFOO_INSTALL_DEFS = {
    # adds "--option=value" to the command
    '--option': 'value',
}

LIBFOO_INSTALL_ENV¶

Provides a means to pass environment variables into the installation process. This option is defined as a dictionary with key-value pairs where the key is the environment name and the value is the environment variable’s value. This field is optional.

LIBFOO_INSTALL_ENV = {
    'OPTION': 'VALUE',
}

LIBFOO_INSTALL_OPTS¶

Provides a means to pass command line options into the installation process. This option can be defined as a dictionary of string pairs or a list with strings – either way defined will generate argument values to include in the installation event. This field is optional.

LIBFOO_INSTALL_OPTS = {
    # adds "--option value" to the command
    '--option': 'value',
}

# (or)

LIBFOO_INSTALL_OPTS = [
    # adds "--some-option" to the command
    '--some-option',
]

CMake Package¶

A CMake package provides support for processing a CMake supported module.

LIBFOO_TYPE = 'cmake'

During the configuration stage of a CMake package, cmake will be invoked to generate build files for the module. For the build stage, cmake --build will be invoked to generated build files. Similar approach for the installation stage where the build option is invoked again but with the install target invoked: cmake --build --target install. Each stage can be configured to manipulate environment variables and options used by the CMake executable.

The default configuration built for projects is RelWithDebInfo. A developer can override this option by explicitly adjusting the configuration option --config to, for example, Debug:

LIBFOO_CONF_DEFS = {
   'CMAKE_BUILD_TYPE': 'Debug',
}

LIBFOO_BUILD_OPTS = {
   '--config': 'Debug',
}

LIBFOO_INSTALL_OPTS = {
   '--config': 'Debug',
}

The following sections outline configuration options are available for a CMake package.

LIBFOO_BUILD_DEFS¶

Provides a means to pass definitions into the build process. This option can is defined as a dictionary of string pairs. This field is optional.

LIBFOO_BUILD_DEFS = {
    # adds "--option=value" to the command
    '--option': 'value',
}

LIBFOO_BUILD_ENV¶

LIBFOO_BUILD_ENV = {
    'OPTION': 'VALUE',
}

LIBFOO_BUILD_OPTS¶

LIBFOO_BUILD_OPTS = {
    # adds "--option value" to the command
    '--option': 'value',
}

# (or)

LIBFOO_BUILD_OPTS = [
    # adds "--some-option" to the command
    '--some-option',
]

LIBFOO_CMAKE_NOINSTALL¶

Specifies whether or not the CMake package should skip an attempt to invoke the install command. Ideally, projects will have an install rule configured to define how a project will install files into a target (or staging) environment. Not all CMake projects may have this rule defined, which can cause the installation stage for a package to fail. A developer can specify this no-install flag to skip a CMake-driven install request and manage installation actions through other means (such as post-processing). By default, the installation stage is invoked with a value of False.

LIBFOO_CMAKE_NOINSTALL = True

LIBFOO_CONF_DEFS¶

Provides a means to pass definitions into the configuration process. This option can is defined as a dictionary of string pairs. This field is optional.

LIBFOO_CONF_DEFS = {
    # adds "--option=value" to the command
    '--option': 'value',
}

LIBFOO_CONF_ENV¶

LIBFOO_CONF_ENV = {
    'OPTION': 'VALUE',
}

LIBFOO_CONF_OPTS¶

LIBFOO_CONF_OPTS = {
    # adds "--option value" to the command
    '--option': 'value',
}

# (or)

LIBFOO_CONF_OPTS = [
    # adds "--some-option" to the command
    '--some-option',
]

LIBFOO_INSTALL_DEFS¶

Provides a means to pass definitions into the installation process. This option can is defined as a dictionary of string pairs. This field is optional.

LIBFOO_INSTALL_DEFS = {
    # adds "--option=value" to the command
    '--option': 'value',
}

LIBFOO_INSTALL_ENV¶

LIBFOO_INSTALL_ENV = {
    'OPTION': 'VALUE',
}

LIBFOO_INSTALL_OPTS¶

LIBFOO_INSTALL_OPTS = {
    # adds "--option value" to the command
    '--option': 'value',
}

# (or)

LIBFOO_INSTALL_OPTS = [
    # adds "--some-option" to the command
    '--some-option',
]

Make Package¶

A make package provides support to easily invoke GNU Make commands at various stages of a package.

LIBFOO_TYPE = 'make'

Make-based projects by default will invoke the default target during the build stage, and invoke the install target for the installation stage. Developers can configure a specific target to invoke during the build stage by specifying a LIBFOO_BUILD_OPTS configuration. For example, if a package uses the target release for standard release builds, the following can be used:

LIBFOO_BUILD_OPTS = [
    'release',
]

For the installation stage, the install target is typically invoked. However, developers can override what target to invoke by adding it into the install options:

LIBFOO_INSTALL_OPTS = [
    'install-minimal',
]

For packages which do not have an installation target to run, developers can use the LIBFOO_MAKE_NOINSTALL option to skip the installation stage for a package.

Default configurations for a make package will not run a configuration stage. However, if a user wants to run a specific target during this stage, the target can be added into the configuration options. For example, if the Makefile configuration has a target prework that should be invoked during the configuration stage, the following can be used:

LIBFOO_CONF_OPTS = [
    'prework',
]

Alternatively, if no configuration options are specified, a <package>-configure script can be invoked if available.

The following sections outline configuration options are available for a make package.

LIBFOO_BUILD_DEFS¶

Provides a means to pass definitions into the build process. This option can is defined as a dictionary of string pairs. This field is optional.

LIBFOO_BUILD_DEFS = {
    # adds "--option=value" to the command
    '--option': 'value',
}

LIBFOO_BUILD_ENV¶

LIBFOO_BUILD_ENV = {
    'OPTION': 'VALUE',
}

LIBFOO_BUILD_OPTS¶

LIBFOO_BUILD_OPTS = {
    # adds "--option value" to the command
    '--option': 'value',
}

# (or)

LIBFOO_BUILD_OPTS = [
    # adds "--some-option" to the command
    '--some-option',
]

LIBFOO_CONF_DEFS¶

Provides a means to pass definitions into the configuration process. This option can is defined as a dictionary of string pairs. This field is optional.

LIBFOO_CONF_DEFS = {
    # adds "--option=value" to the command
    '--option': 'value',
}

LIBFOO_CONF_ENV¶

LIBFOO_CONF_ENV = {
    'OPTION': 'VALUE',
}

LIBFOO_CONF_OPTS¶

LIBFOO_CONF_OPTS = {
    # adds "--option value" to the command
    '--option': 'value',
}

# (or)

LIBFOO_CONF_OPTS = [
    # adds "--some-option" to the command
    '--some-option',
]

LIBFOO_INSTALL_DEFS¶

Provides a means to pass definitions into the installation process. This option can is defined as a dictionary of string pairs. This field is optional.

LIBFOO_INSTALL_DEFS = {
    # adds "--option=value" to the command
    '--option': 'value',
}

LIBFOO_INSTALL_ENV¶

LIBFOO_INSTALL_ENV = {
    'OPTION': 'VALUE',
}

LIBFOO_INSTALL_OPTS¶

LIBFOO_INSTALL_OPTS = {
    # adds "--option value" to the command
    '--option': 'value',
}

# (or)

LIBFOO_INSTALL_OPTS = [
    # adds "--some-option" to the command
    '--some-option',
]

LIBFOO_MAKE_NOINSTALL¶

Specifies whether a make package should skip an attempt to invoke the install target. Ideally, projects will have an install target configured to define how a project will install files into a target (or staging) environment. Not all make projects may have this target defined, which can cause the installation stage for a package to fail. A developer can specify this no-install flag to skip a make install target request and manage installation actions through other means (such as post-processing). By default, the installation stage is invoked with a value of False.

LIBFOO_MAKE_NOINSTALL = True

Python Package¶

A Python package provides support for processing a Python supported module.

LIBFOO_TYPE = 'python'

Only the build and installation phases are used when processing the sources for a Python package (i.e. no configuration stage is invoked). The build phase will invoke setup.py build while the installation stage will invoke setup.py install. When a Python package is process, it will use the system’s default Python interpreter. A developer can override what Python interpreter to use by configuring the PYTHON_INTERPRETER option in a package:

LIBFOO_PYTHON_INTERPRETER = '/opt/my-custom-python-build/python'

The following sections outline configuration options are available for a Python package.

LIBFOO_BUILD_DEFS¶

Provides a means to pass definitions into the build process. This option can is defined as a dictionary of string pairs. This field is optional.

LIBFOO_BUILD_DEFS = {
    # adds "--option=value" to the command
    '--option': 'value',
}

LIBFOO_BUILD_ENV¶

LIBFOO_BUILD_ENV = {
    'OPTION': 'VALUE',
}

LIBFOO_BUILD_OPTS¶

LIBFOO_BUILD_OPTS = {
    # adds "--option value" to the command
    '--option': 'value',
}

# (or)

LIBFOO_BUILD_OPTS = [
    # adds "--some-option" to the command
    '--some-option',
]

LIBFOO_INSTALL_DEFS¶

Provides a means to pass definitions into the installation process. This option can is defined as a dictionary of string pairs. This field is optional.

LIBFOO_INSTALL_DEFS = {
    # adds "--option=value" to the command
    '--option': 'value',
}

LIBFOO_INSTALL_ENV¶

LIBFOO_INSTALL_ENV = {
    'OPTION': 'VALUE',
}

LIBFOO_INSTALL_OPTS¶

LIBFOO_INSTALL_OPTS = {
    # adds "--option value" to the command
    '--option': 'value',
}

# (or)

LIBFOO_INSTALL_OPTS = [
    # adds "--some-option" to the command
    '--some-option',
]

LIBFOO_PYTHON_INTERPRETER¶

Defines a specific Python interpreter when processing the build and installation stages for a package. If not specified, the system’s Python interpreter will be used. This field is optional.

LIBFOO_PYTHON_INTERPRETER = '<path>'

LIBFOO_PYTHON_SETUP_TYPE¶

The setup type will configure how a Python package is built and installed. The default setup type used for a Python package is a distutils package. It is recommended to always configure a setup type for a Python package. The following outlines the available setup types in releng-tool:

Type	Value
Flit	`flit`
Hatch	`hatch`
PDM	`pdm`
PEP 517 build	`pep517`
Poetry	`poetry`
Setuptools	`setuptools`
distutils	`distutils`

For example:

LIBFOO_PYTHON_SETUP_TYPE = 'setuptools'

For setup types other than Setuptools/distutils, the installer module will be used to install packages to their destination folders.

Host environments are required to pre-install needed packages in their running Python environment to support setup types not available in a standard Python distribution. For example, if a PDM setup type is set, the host system will need to have pdm Python module installed on the system.

SCons Package¶

A SCons package provides support to easily invoke SCons commands at various stages of a package.

LIBFOO_TYPE = 'scons'

SCons-based projects by default will invoke the default target during the build stage, and invoke the install alias for the installation stage. Developers can configure a specific target to invoke during the build stage by specifying a LIBFOO_BUILD_OPTS configuration. For example, if a package uses the target release for standard release builds, the following can be used:

LIBFOO_BUILD_OPTS = [
    'release',
]

For the installation stage, the install alias is typically invoked. However, developers can override what target to invoke by adding it into the install options:

LIBFOO_INSTALL_OPTS = [
    'install-minimal',
]

For packages which do not have an installation alias to run, developers can use the LIBFOO_SCONS_NOINSTALL option to skip the installation stage for a package.

Default configurations for a SCons package will not run a configuration stage. However, if a user wants to run a specific target during this stage, the target can be added into the configuration options. For example, if the SCons definition has a target prework that should be invoked during the configuration stage, the following can be used:

LIBFOO_CONF_OPTS = [
    'prework',
]

Alternatively, if no configuration options are specified, a <package>-configure script can be invoked if available.

The following sections outline configuration options are available for a SCons package.

LIBFOO_BUILD_DEFS¶

Provides a means to pass definitions into the build process. This option can is defined as a dictionary of string pairs. This field is optional.

LIBFOO_BUILD_DEFS = {
    # adds "--option=value" to the command
    '--option': 'value',
}

LIBFOO_BUILD_ENV¶

LIBFOO_BUILD_ENV = {
    'OPTION': 'VALUE',
}

LIBFOO_BUILD_OPTS¶

LIBFOO_BUILD_OPTS = {
    # adds "--option value" to the command
    '--option': 'value',
}

# (or)

LIBFOO_BUILD_OPTS = [
    # adds "--some-option" to the command
    '--some-option',
]

LIBFOO_CONF_DEFS¶

Provides a means to pass definitions into the configuration process. This option can is defined as a dictionary of string pairs. This field is optional.

LIBFOO_CONF_DEFS = {
    # adds "--option=value" to the command
    '--option': 'value',
}

LIBFOO_CONF_ENV¶

LIBFOO_CONF_ENV = {
    'OPTION': 'VALUE',
}

LIBFOO_CONF_OPTS¶

LIBFOO_CONF_OPTS = {
    # adds "--option value" to the command
    '--option': 'value',
}

# (or)

LIBFOO_CONF_OPTS = [
    # adds "--some-option" to the command
    '--some-option',
]

LIBFOO_INSTALL_DEFS¶

Provides a means to pass definitions into the installation process. This option can is defined as a dictionary of string pairs. This field is optional.

LIBFOO_INSTALL_DEFS = {
    # adds "--option=value" to the command
    '--option': 'value',
}

LIBFOO_INSTALL_ENV¶

LIBFOO_INSTALL_ENV = {
    'OPTION': 'VALUE',
}

LIBFOO_INSTALL_OPTS¶

LIBFOO_INSTALL_OPTS = {
    # adds "--option value" to the command
    '--option': 'value',
}

# (or)

LIBFOO_INSTALL_OPTS = [
    # adds "--some-option" to the command
    '--some-option',
]

LIBFOO_SCONS_NOINSTALL¶

Specifies whether a SCons package should skip an attempt to invoke the install alias. Ideally, projects will have an install alias defined to specify how a project will install files into a target (or staging) environment. Not all SCons projects may have this target defined, which can cause the installation stage for a package to fail. A developer can specify this no-install flag to skip a SCons install target request and manage installation actions through other means (such as post-processing). By default, the installation stage is invoked with a value of False.

LIBFOO_SCONS_NOINSTALL = True

Other¶

Post-processing¶

Warning

A post-processing script (if used) will be invoked each time releng-tool reaches the final stage of a build.

After each package has been processed, a project has the ability to perform post-processing. Post-processing allows a developer to cleanup the target directory, build an archive/package from generated results and more. If a project contains a releng-post-build inside the root directory, the post-processing script will be invoked in the final stage of a build.

A developer may start out with the following post-processing script <root>/releng-post-build:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

print('post processing...')

The above script will output the newly inserted print message at the end of a build process:

$ releng-tool
...
generating license information...
post processing...
(success) completed (0:01:30)

A developer can take advantage of environment variables and script helpers for additional support.

It is important to note that a post-processing script will be invoked each time a releng-tool invoke reaches the final stage of a build. A developer should attempt to implement the post-processing script in a way that it can be invoked multiple times. For example, if a developer decides to move a file out of the target directory into an interim directory when building an archive, it is most likely that a subsequent request to build may fail since the file can no longer be found inside the target directory.

Licenses¶

A releng-tool project can defined multiple packages, each with the possibility of having multiple licenses associated with them. Each project may vary: some may have only proprietary sources and may not care about tracking this information; some may only use open source software and require to populate license information for a final package; or a mix.

When license information is populated for a project, each project’s license information (LIBFOO_LICENSE_FILES) is will be populated into a single license document. If a developer defines the license_header configuration, the generated document will be prefixed with the header content. For example, releng can be configured to prepare a license header from a local file assets/license-header.tpl:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import os

... (other configuration options)

root_dir = os.path.dirname(os.path.realpath(__file__))
license_header_file = os.path.join(root_dir, 'assets', 'license-header.tpl')

with open(license_header_file) as f:
    license_header = ''.join(f.readlines())

A side note is that licenses for a project are generated before the post-processing phase; hence, generated license document(s) may be included when attempting to generated final archives/packages.

Patching¶

Note

Patches are ignored when in development mode for packages with a development revision, for local VCS packages or when in local-sources mode for internal packages.

The patching stage for a package provides the ability for a developer to apply one or more patches to extracted sources. A project may define an external package which fetches an archive that is not maintained by the project owners. The fetched source may not be able to build in a developer’s releng-tool project due to limitations of the implementation or build scripts provided by the package. A developer can prepare a series of patches to apply to a package and submit changes upstream to correct the issue; however, the developer is then left to either wait for the changes to be merged in or needs to make a custom archive with the appropriate modifications already applied. To avoid this, a developer can include patches directly in the package folder to be automatically applied during patching stage.

When a package’s patch stage is reached, releng-tool will look for patches found inside the package folder with the extension .patch. Patches found inside a package folder are applied in ascending order. It is recommended to prefix patch filenames with a numerical value for clarity. For example, the following package patches:

$ cd package/liba
$ ls *.patch
0001-accept-linker-flags.patch
0002-correct-output-path.patch
0003-support-disabling-test-build.patch
liba
liba.hash

With be applied in the following order:

0001-accept-linker-flags.patch
0002-correct-output-path.patch
0003-support-disabling-test-build.patch

If a user configures their build environment in development mode, patches will not be applied if a package defines a development revisions. The idea is that a development revision is most likely the bleeding edge source of a package and does not need any patches. If a user configures their build environment in local-sources mode and a package is defined as internal, patches will not be applied to the sources. This is to prevent the patching system from making unexpected modifications to a developer’s local source variants.

Internal and External packages¶

Packages are either internal packages or external packages. All packages are considered external packages by default unless explicitly configured as internal through either the package option LIBFOO_INTERNAL or using the project configuration default_internal (see also LIBFOO_EXTERNAL). Both package types are almost treated the same except for the following:

An internal package will not generate output warnings if the package is missing hash information or an ASCII-armor.
An internal package will not generate output warnings if the package is missing license information.
When configured for development mode; the patch stage will not be performed if the package specifies a development revision (LIBFOO_DEVMODE_REVISION).
When configured for local-sources mode; the fetch, extract and patch stages will not be performed.

Script helpers¶

releng-tool provides a series of helper functions which can be used in script-based packages, post-processing and more. Helper functions provided are listed below:

Method	Documentation
`debug`	releng_tool.debug(msg, args) Logs a debug message to standard out with a trailing new line. By default, debug messages will not be output to standard out unless the instance is configured with debugging enabled. debug('this is a debug message') Parameters: msg* – the message *args – an arbitrary set of positional and keyword arguments used when generating a formatted message
`err`	releng_tool.err(msg, args) Logs an error message to standard error with a trailing new line and (if enabled) a red colorization. err('this is an error message') Parameters: msg* – the message *args – an arbitrary set of positional and keyword arguments used when generating a formatted message
`hint`	releng_tool.hint(msg, args) Logs a hint message to standard out with a trailing new line and (if enabled) a cyan colorization. hint('this is a hint message') Parameters: msg* – the message *args – an arbitrary set of positional and keyword arguments used when generating a formatted message
`log`	releng_tool.log(msg, args) Logs a (normal) message to standard out with a trailing new line. log('this is a message') Parameters: msg* – the message *args – an arbitrary set of positional and keyword arguments used when generating a formatted message
`note`	releng_tool.note(msg, args) Logs a notification message to standard out with a trailing new line and (if enabled) an inverted colorization. note('this is a note message') Parameters: msg* – the message *args – an arbitrary set of positional and keyword arguments used when generating a formatted message
`releng_cat`	releng_tool.releng_cat(file, args) Attempts to read one or more files provided to this call. For each file, it will be read and printed out to the standard output. An example when using in the context of script helpers is as follows: releng_cat('my-file') Parameters: file* – the file args (optional*) – additional files to include Returns: `True` if all the files exists and were printed to the standard output; `False` if one or more files could not be read
`releng_copy`	releng_tool.releng_copy(src, dst, quiet=False, critical=True, dst_dir=None) This call will attempt to copy a provided file or directory, defined by `src` into a destination file or directory defined by `dst`. If `src` is a file, then `dst` is considered to be a file or directory; if `src` is a directory, `dst` is considered a target directory. If a target directory or target file’s directory does not exist, it will be automatically created. In the event that a file or directory could not be copied, an error message will be output to standard error (unless `quiet` is set to `True`). If `critical` is set to `True` and the specified file/directory could not be copied for any reason, this call will issue a system exit (`SystemExit`). An example when using in the context of script helpers is as follows: # (stage) # my-file releng_copy('my-file', 'my-file2') # (stage) # my-file # my-file2 releng_copy('my-file', 'my-directory/') # (stage) # my-directory/my-file # my-file # my-file2 releng_copy('my-directory/', 'my-directory2/') # (stage) # my-directory/my-file # my-directory2/my-file # my-file # my-file2 Parameters: src – the source directory or file dst – the destination directory or file* (if `src` is a file) quiet* (optional) – whether or not to suppress output critical (optional) – whether or not to stop execution on failure dst_dir (optional) – force hint that the destination is a directory Returns: `True` if the copy has completed with no error; `False` if the copy has failed Raises: SystemExit – if the copy operation fails with `critical=True`
`releng_copy_into`	releng_tool.releng_copy_into(src, dst, quiet=False, critical=True) This call will attempt to copy a provided file or directory, defined by `src` into a destination directory defined by `dst`. If a target directory does not exist, it will be automatically created. In the event that a file or directory could not be copied, an error message will be output to standard error (unless `quiet` is set to `True`). If `critical` is set to `True` and the specified file/directory could not be copied for any reason, this call will issue a system exit (`SystemExit`). An example when using in the context of script helpers is as follows: # (stage) # my-file releng_copy_into('my-file', 'my-directory') # (stage) # my-directory/my-file # my-file releng_copy_into('my-directory', 'my-directory2') # (stage) # my-directory/my-file # my-directory2/my-file # my-file Parameters: src – the source directory or file dst – the destination directory quiet (optional) – whether or not to suppress output critical (optional) – whether or not to stop execution on failure Returns: `True` if the copy has completed with no error; `False` if the copy has failed Raises: SystemExit – if the copy operation fails with `critical=True`
`releng_env`	releng_tool.releng_env(key, value=<object object>) Provides a caller a simple method to fetch or configure an environment variable for the current context. This call is the same as if one directly fetched from or managed a key-value with `os.environ`. If `value` is not provided, the environment variable’s value (if set) will be returned. If `value` is set to a value of `None`, any set environment variable will be removed. An example when using in the context of script helpers is as follows: # get an environment variable value = releng_env('KEY') # set an environment variable releng_env('KEY', 'VALUE') Parameters: key – the environment key value (optional) – the environment value to set Returns: the value of the environment variable
`releng_execute`	releng_tool.releng_execute(args, cwd=None, env=None, env_update=None, quiet=None, critical=True, poll=False, capture=None) Runs the command described by `args` until completion. A caller can adjust the working directory of the executed command by explicitly setting the directory in `cwd`. The execution request will return `True` on a successful execution; `False` if an issue has been detected (e.g. bad options or called process returns a non-zero value). In the event that the execution fails, an error message will be output to standard error unless `quiet` is set to `True`. The environment variables used on execution can be manipulated in two ways. First, the environment can be explicitly controlled by applying a new environment content using the `env` dictionary. Key of the dictionary will be used as environment variable names, whereas the respective values will be the respective environment variable’s value. If `env` is not provided, the existing environment of the executing context will be used. Second, a caller can instead update the existing environment by using the `env_update` option. Like `env`, the key-value pairs match to respective environment key-value pairs. The difference with this option is that the call will use the original environment values and update select values which match in the updated environment request. When `env` and `env_update` are both provided, `env_update` will be updated the options based off of `env` instead of the original environment of the caller. If `critical` is set to `True` and the execution fails for any reason, this call will issue a system exit (`SystemExit`). By default, the critical flag is enabled (i.e. `critical=True`). In special cases, an executing process may not provide carriage returns/new lines to simple output processing. This can lead the output of a process to be undesirably buffered. To workaround this issue, the execution call can instead poll for output results by using the `poll` option with a value of `True`. By default, polling is disabled with a value of `False`. A caller may wish to capture the provided output from a process for examination. If a list is provided in the call argument `capture`, the list will be populated with the output provided from an invoked process. An example when using in the context of script helpers is as follows: releng_execute(['echo', '$TEST'], env={'TEST': 'this is a test'}) Parameters: args – the list of arguments to execute cwd (optional) – working directory to use env (optional) – environment variables to use for the process env_update (optional) – environment variables to append for the process quiet (optional) – whether or not to suppress output (defaults to `False`) critical (optional) – whether or not to stop execution on failure (defaults to `True`) poll (optional) – force polling stdin/stdout for output data (defaults to `False`) capture (optional) – list to capture output into Returns: `True` if the execution has completed with no error; `False` if the execution has failed Raises: SystemExit – if the execution operation fails with `critical=True`
`releng_execute_rv`	releng_tool.releng_execute_rv(command, args, kwargs) Runs the command `command` with provided `args` until completion. A caller can adjust the working directory of the executed command by explicitly setting the directory in `cwd`. The execution request will return the command’s return code as well as any captured output. The environment variables used on execution can be manipulated in two ways. First, the environment can be explicitly controlled by applying a new environment content using the `env` dictionary. Key of the dictionary will be used as environment variable names, whereas the respective values will be the respective environment variable’s value. If `env` is not provided, the existing environment of the executing context will be used. Second, a caller can instead update the existing environment by using the `env_update` option. Like `env`, the key-value pairs match to respective environment key-value pairs. The difference with this option is that the call will use the original environment values and update select values which match in the updated environment request. When `env` and `env_update` are both provided, `env_update` will be updated the options based off of `env` instead of the original environment of the caller. An example when using in the context of script helpers is as follows: rv, out = releng_execute_rv('echo', '$TEST', env={'TEST': 'env-test'}) Parameters: command* – the command to invoke args (optional) – arguments to add to the command cwd – working directory to use env – environment variables to use for the process *env_update – environment variables to append for the process Returns: the return code and output of the execution request
`releng_exists`	releng_tool.releng_exists(path, args) Allows a caller to verify the existence of a file on the file system. This call will return `True` if the path exists; `False` otherwise. An example when using in the context of script helpers is as follows: if releng_exists('my-file'): print('the file exists') else: print('the file does not exist') Parameters: path* – the path *args – additional path parts Returns: `True` if the path exists; `False` otherwise
`releng_exit`	releng_tool.releng_exit(msg=None, code=None) Provides a convenience method to help invoke a system exit call without needing to explicitly use `sys`. A caller can provide a message to indicate the reason for the exit. The provide message will output to standard error. The exit code, if not explicit set, will vary on other arguments. If a message is provided to this call, the default exit code will be `1`. If no message is provided, the default exit code will be `0`. In any case, if the caller explicitly sets a code value, the provided code value will be used. An example when using in the context of script helpers is as follows: releng_exit('there was an error performing this task') Parameters: msg (optional) – error message to print code (optional) – exit code; defaults to 0 if no message or defaults to 1 if a message is set Raises: SystemExit – always raised
`releng_expand`	releng_tool.releng_expand(obj, kv=None) This expand utility method will attempt to expand variables in detected string types. For a detected string which contains substrings in the form of `$value` or `${value}`, these substrings will be replaced with their respective key-value (if provided) or environment variable value. For substrings which do not have a matching variable value, the substrings will be replaced with an empty value. If a dictionary is provided, keys and values will be checked if they can be expanded on. If a list/set is provided, each value which be checked if it can be expanded on. If a dictionary key is expanded to match another key, a key-value pair can be dropped. If a set may result in a smaller set if expanded values result in duplicate entries. An example when using in the context of script helpers is as follows: import os ... os.environ['MY_ENV'] = 'my-environment-variable' value = releng_expand('$MY_ENV') print(value) # will output: my-environment-variable Parameters: obj – the object kv (optional) – key-values pairs to use Returns: the expanded object
`releng_include`	releng_tool.releng_include(file_path) The provided call will execute code at the provided file path. The path will be relative to the caller’s script, unless an absolute path is provided. The executed script will be initialized with globals matching the caller’s script. An example when using in the context of script helpers is as follows: # load "my-other-script" found alongside the current script releng_include('my-other-script') Parameters: file_path – the script to invoke
`releng_join`	releng_tool.releng_join(path, *paths)¶ An alias for `os.path.join`.
`releng_ls`	releng_tool.releng_ls(dir_) Attempts to read a directory for its contents and prints this information to the configured standard output stream. An example when using in the context of script helpers is as follows: releng_ls('my-dir/') Parameters: dir – the directory Returns: `True` if the directory could be read and its contents have been printed to the standard output; `False` if the directory could not be read
`releng_mkdir`	releng_tool.releng_mkdir(dir_, quiet=False, critical=False) Attempts to create the provided directory. If the directory already exists, this method has no effect. If the directory does not exist and could not be created, this method will return `False`. Also, if an error has been detected, an error message will be output to standard error (unless `quiet` is set to `True`). An example when using in the context of script helpers is as follows: if releng_mkdir('my-directory'): print('directory was created') else: print('directory was not created') Parameters: dir – the directory quiet (optional) – whether or not to suppress output (defaults to `False`) critical (optional) – whether or not to stop execution on failure (defaults to `False`) Returns: `True` if the directory exists; `False` if the directory could not be created
`releng_move`	releng_tool.releng_move(src, dst, quiet=False, critical=True, dst_dir=None) This call will attempt to move a provided file or directory’s contents, defined by `src` into a destination file or directory defined by `dst`. If `src` is a file, then `dst` is considered to be a file or directory; if `src` is a directory, `dst` is considered a target directory. If a target directory or target file’s directory does not exist, it will be automatically created. In the event that a file or directory could not be moved, an error message will be output to standard error (unless `quiet` is set to `True`). If `critical` is set to `True` and the specified file/directory could not be moved for any reason, this call will issue a system exit (`SystemExit`). An example when using in the context of script helpers is as follows: # (input) # my-directory/another-file # my-file # my-file2 releng_move('my-file', 'my-file3') releng_move('my-directory/', 'my-directory2/') releng_move('my-file2', 'my-directory2/') # (output) # my-directory2/another-file # my-directory2/my-file2 # my-file3 Parameters: src – the source directory or file dst – the destination directory or file* (if `src` is a file) quiet* (optional) – whether or not to suppress output critical (optional) – whether or not to stop execution on failure dst_dir (optional) – force hint that the destination is a directory Returns: `True` if the move has completed with no error; `False` if the move has failed Raises: SystemExit – if the copy operation fails with `critical=True`
`releng_move_into`	releng_tool.releng_move_into(src, dst, quiet=False, critical=True) This call will attempt to move a provided file or directory’s contents, defined by `src` into a destination directory defined by `dst`. If a target directory directory does not exist, it will be automatically created. In the event that a file or directory could not be moved, an error message will be output to standard error (unless `quiet` is set to `True`). If `critical` is set to `True` and the specified file/directory could not be moved for any reason, this call will issue a system exit (`SystemExit`). An example when using in the context of script helpers is as follows: # (input) # my-directory/another-file # my-file # my-file2 releng_move('my-file', 'my-file3') releng_move('my-directory', 'my-directory2') releng_move('my-file2', 'my-directory2') # (output) # my-directory2/another-file # my-directory2/my-file2 # my-file3 Parameters: src – the source directory or file dst – the destination directory quiet (optional) – whether or not to suppress output critical (optional) – whether or not to stop execution on failure Returns: `True` if the move has completed with no error; `False` if the move has failed Raises: SystemExit – if the copy operation fails with `critical=True`
`releng_remove`	releng_tool.releng_remove(path, quiet=False) Attempts to remove the provided path if it exists. The path value can either be a directory or a specific file. If the provided path does not exist, this method has no effect. In the event that a file or directory could not be removed due to an error other than unable to be found, an error message will be output to standard error (unless `quiet` is set to `True`). An example when using in the context of script helpers is as follows: releng_remove('my-file') # (or) releng_remove('my-directory/') Parameters: path – the path to remove quiet (optional) – whether or not to suppress output Returns: `True` if the path was removed or does not exist; `False` if the path could not be removed from the system
`releng_require_version`	releng_tool.releng_require_version(version, quiet=False, critical=True) Enables a caller to explicitly check for a required releng-tool version. Invoking this function with a dotted-separated `version` string, the string will be parsed and compared with the running releng-tool version. If the required version is met, this method will have no effect. In the event that the required version is not met, the exception `SystemExit` will be raised if the critical flag is set; otherwise this call will return `False`. An example when using in the context of script helpers is as follows: # ensure we are using releng-tool v1 releng_require_version('1.0.0') Parameters: version – dotted-separated version string quiet (optional) – whether or not to suppress output critical (optional) – whether or not to stop execution on failure Returns: `True` if the version check is met; `False` if the version check has failed Raises: SystemExit – if the version check fails with `critical=True`
`releng_tmpdir`	releng_tool.releng_tmpdir(dir_=None) Creates a temporary directory in the provided directory `dir_` (or system default, is not provided). This is a context-supported call and will automatically remove the directory when completed. If the provided directory does not exist, it will created. If the directory could not be created, an `FailedToPrepareBaseDirectoryError` exception will be thrown. An example when using in the context of script helpers is as follows: with releng_tmpdir() as dir_: print(dir_) Parameters: dir (optional) – the directory to create the temporary directory in Raises: FailedToPrepareBaseDirectoryError – the base directory does not exist and could not be created
`releng_touch`	releng_tool.releng_touch(file) Attempts to update the access/modifications times on a file. If the file does not exist, it will be created. This utility call operates in the same fashion as the `touch` system command. An example when using in the context of script helpers is as follows: if releng_touch('my-file'): print('file was created') else: print('file was not created') Parameters: file – the file Returns: `True` if the file was created/updated; `False` if the file could not be created/updated
`releng_wd`	releng_tool.releng_wd(dir_) Moves the current context into the provided working directory `dir`. When returned, the original working directory will be restored. If the provided directory does not exist, it will created. If the directory could not be created, an `FailedToPrepareWorkingDirectoryError` exception will be thrown. An example when using in the context of script helpers is as follows: with releng_wd('my-directory/'): # invoked in 'my-directory' # invoked in original working directory Parameters: dir – the target working directory Raises: FailedToPrepareWorkingDirectoryError – the working directory does not exist and could not be created
`success`	releng_tool.success(msg, args) Logs a success message to standard error with a trailing new line and (if enabled) a green colorization. success('this is a success message') Parameters: msg* – the message *args – an arbitrary set of positional and keyword arguments used when generating a formatted message
`verbose`	releng_tool.verbose(msg, args) Logs a verbose message to standard out with a trailing new line and (if enabled) an inverted colorization. By default, verbose messages will not be output to standard out unless the instance is configured with verbosity. verbose('this is a verbose message') Parameters: msg* – the message *args – an arbitrary set of positional and keyword arguments used when generating a formatted message
`warn`	releng_tool.warn(msg, args) Logs a warning message to standard error with a trailing new line and (if enabled) a purple colorization. warn('this is a warning message') Parameters: msg* – the message args – an arbitrary set of positional and keyword arguments used when generating a formatted message Raises: RelengToolWarningAsError* – when warnings-are-errors is configured

Scripts directly invoked by releng-tool will automatically have these helpers registered in the script’s globals module (i.e. no import is necessary). If a project defines custom Python modules in their project and wishes to take advantage of these helper functions, the following import can be used to, for example, import a specific function:

from releng_tool import releng_execute

Or, if desired, all helper methods can be imported at once:

from releng_tool import *

VCS Ignore¶

When invoking releng-tool on a project, the project’s root directory will be populated with cached assets and other output files. A series of standard ignore patterns can be applied to a repository to prevent observing these generated files using VCS tools. The following is an example ignore configuration which can be applied for Git-based repositories (via .gitignore):

# releng-tool
/cache/
/dl/
/output/
.releng-flag-*

Quirks¶

The following outlines a series of configuration quirks that can be applied to deal with rare host environment scenarios.

releng.bzr.certifi – Use certifi certificates for bzr exports.
releng.cmake.disable_direct_includes – Disables injecting include directories into CMake configuration stages for host, staging and target include directories (when applicable). This can be used to prevent possible issues when building a CMake project with a toolchain file which has issues with standard include overrides from releng-tool.
releng.cmake.disable_parallel_option – Disables the --parallel option from being injected when a CMake-package is performing a build stage. This is to help support host systems running variants of CMake which do not explicitly provide a parallelization option.
releng.disable_prerequisites_check – Disables the fast-fail prerequisites check from running.
releng.disable_remote_configs – Disables any attempt to load remote-defined configurations for a package.
releng.disable_remote_scripts – Disables any attempt to load remote-defined scripts for a package.
releng.disable_spdx_check – Disables any attempt to compare package licenses to SPDX license identifiers.
releng.git.no_depth – Disables the --depth option from being injected when a Git-site-defined package is performing a fetch stage.
releng.git.no_quick_fetch – Disables quick fetching (a specific reference fetch) for any Git-site-defined package.

Quirks are enabled through a configuration (override) script such as follows:

quirks = [
    '<quirk-value>',
]

Loading extensions¶

Note

If looking for information on developing extensions for releng-tool, consult the contributor’s guide – extensions.

A releng-tool project can define one or more extensions to load for externally implemented capabilities. For example, a project can load extensions ext-a and ext-b with the following defined in their project’s configuration:

extensions = [
    'ext-a',
    'ext-b',
]

During the initial stages of a releng-tool process, the process will check and load any configured extension. In the event that an extension is missing, is unsupported for the running releng-tool version or fails to load, a detailed error message will be presented to the user.

While the ability to load extensions is supported, capabilities provided by extensions are not officially supported by releng-tool. For issues related to specific extension use, it is recommended to see the documentation provided by the providers of said extensions.