Understanding fetching

The first stage packages go through is the “fetch” phase. For packages which have content to acquire, files can be downloaded or version control sources may be locally cached. For example, if a package has a link to a file to download:

LIBFOO_SITE = 'https://example.com/libfoo-1.0.tgz'

The file will be stored in the download folder, ready to be used for extraction:

└── my-project/
    ├── dl/
    │   └── libfoo/
    │       └── libfoo-1.0.tgz       <----
    ├── package/
    │   └── libfoo/
    │       └── ...
    └── releng

And for DVCS-based sites:

LIBFOO_SITE = 'git+git@example.com:base/libfoo.git'

The contents of the repository will be stored in the cache folder for later use:

└── my-project/
    ├── cache/
    │   └── libfoo/
    │       └── ...                  <----
    ├── package/
    │   └── libfoo/
    │       └── ...
    └── releng

Once a package has completed its fetch stage, releng-tool should never need to remotely fetch content for that package again (until site, version, etc. changes). In the event that a user invokes a clean action followed by a full build:

$ releng-tool clean
$ releng-tool
...

No fetching may occur since package contents may already be stored in the existing dl/ and cache/ folders.

A problem a user may experience is if they wish to re-acquire sources from a site if they know the sources have changed. For example, if a remote archive has been updated, if a tag has been moved or if a target branch has been updated. The following will present a series of ways a user can deal with these use cases.

Start fresh!

Tip

While this approach ensures the most recent sources for the existing package configurations are fetched, it can be time consuming to clear the entire cache if, for example, only a single package has changed.

It is recommended to look at all available options to re-fetch content.

The easiest way to ensure all fresh sources are downloaded is to clean all local downloads/cache for a project. This can be achieved using distclean:

releng-tool distclean

This will ensure all downloaded files, cached content, etc. are removed from the local system, ensuring a next-build to download fresh sources.

Full fetching

Most package sources are acquired during the fetch stage. However, some packages define dependencies within their sources. This can require releng-tool to first fetch a defined package’s sources, extract the package, followed by fetching any defined dependencies. Post-fetching will be automatically performed for supported packages (e.g. Cargo) after their extraction stage. Users can invoke the fetch-full action to explicitly process releng-tool’s fetch-post operations:

releng-tool fetch-full

Force re-fetch of DVCS sources

When a DVCS-based package goes through its fetch stage, the contents can be locally stored in the configured cache directory. For a package definition as follows:

LIBFOO_SITE = 'git+git@example.com:base/libfoo.git'
LIBFOO_REVISION = 'v1.0'

A cache of the v1.0 tag will be fetched and stored locally. If a package is cleaned and rebuilt again, releng-tool will referred to the locally cached tag. In the event that the remote site changes the tag location, clean builds will not be using the most recent tag location. If a user knows the reference for a site has been updated, they can explicitly request a <pkg>-fetch on a package which should trigger a forced update from the remote site. For example:

releng-tool libfoo-fetch

The above can recognize the new tag update, and any future clean builds made will use the new reference implementation.

Force re-fetch of fixed sources

For packages which reference a fixed artifact, once a package has completed its fetch stage, the artifact will no longer need to be downloaded again. For example:

LIBFOO_SITE = 'https://example.com/libfoo-1.0.tgz'

The archive libfoo-1.0.tgz will be locally store and used in future builds. In the event where a site’s archive is known to have been changed, a user can force re-fetch these artifacts by using the <pkg>-fetch action along with the -F, --force argument. For example, even if libfoo-1.0.tgz was already downloaded locally, a request as follows will delete the local cache file and re-download it from the configured site:

releng-tool libfoo-fetch --force

Automatically re-fetch development branches

When operating in development mode where a package has a development-specific source based off a branch, it may be preferred to always ensure the most recent sources are fetched. For example, consider libfoo with a development revision main:

LIBFOO_SITE = 'git+git@example.com:base/libfoo.git'
LIBFOO_REVISION = 'v1.0'
LIBFOO_DEVMODE_REVISION = 'main'

With the above configuration, if a builder cleans and rebuilds the project, the originally cache of main would still be used, even if main has new updates on the remote branch:

$ releng-tool
...
~using libfoo hash 1b100c825307730e2027398d70af7c84ce173238~
...
$ releng-tool clean
$ releng-tool
...
~still using libfoo hash 1b100c825307730e2027398d70af7c84ce173238~
...

If a user wants to automatically re-fetch new updates on a development branch, they can take advantage of the LIBFOO_DEVMODE_IGNORE_CACHE option. For example, if a site definition had:

LIBFOO_SITE = 'git+git@example.com:base/libfoo.git'
LIBFOO_REVISION = 'v1.0'
LIBFOO_DEVMODE_REVISION = 'main'
LIBFOO_DEVMODE_IGNORE_CACHE = True

A repeat of the rebuild actions will result in an automatic update of the package’s main branch (if updates are available):

$ releng-tool
...
~using libfoo hash 1b100c825307730e2027398d70af7c84ce173238~
...
$ releng-tool clean
$ releng-tool
...
~using libfoo hash 4cd6375c7464b6bdf166b4f27d5e2fbd937e4ad0~
...