buildstream.git
9 days agoMerge pull request #1741 from ssssam/sam/bst-2-install master
Tristan Van Berkom [Tue, 27 Sep 2022 15:05:56 +0000 (11:05 -0400)] 
Merge pull request #1741 from ssssam/sam/bst-2-install

Update install instructions for final BuildStream 2.0 release

9 days agomain_install.rst: specify dependencies from requirements.in 1741/head
Tristan Van Berkom [Tue, 27 Sep 2022 15:02:06 +0000 (11:02 -0400)] 
main_install.rst: specify dependencies from requirements.in

The requirements.txt file specifies the frozen dependencies used in CI,
the requirements.in are the actual requirements checked by setup.py

2 weeks agodoc/source/main_install.rst: Rework completions instructions
Sam Thursfield [Tue, 20 Sep 2022 12:47:10 +0000 (14:47 +0200)] 
doc/source/main_install.rst: Rework completions instructions

2 weeks agodoc/source/main_install.rst: Dis-recommend `pip install -e`
Sam Thursfield [Tue, 20 Sep 2022 12:46:50 +0000 (14:46 +0200)] 
doc/source/main_install.rst: Dis-recommend `pip install -e`

2 weeks agodoc/source/main_install.rst: Rework as "Install from source" guide
Sam Thursfield [Tue, 30 Aug 2022 11:49:35 +0000 (13:49 +0200)] 
doc/source/main_install.rst: Rework as "Install from source" guide

For rationale, see:
https://github.com/apache/buildstream/pull/1741#issuecomment-1227820050

Changes include:

  * provide a list of host tools required by source plugins and by
    BuildBox - almost no real-world projects can be built unless these
    are installed.
  * clarify which dependencies are needed only at install time
  * recommend use of latest BuildBox tag, not specific outdated tags
  * remove "install from PyPI" instructions - this belongs on the
    website
  * remove container install docs - these belong on website
  * add info about Bash completions
  * BuildStream 2 is now "in beta", this sounds more complete than "dev
    snapshot".
  * avoid "simply", nothing in tech is ever simple :)

2 weeks agodoc/source/core_plugins.rst: List buildstream-plugins
Sam Thursfield [Tue, 30 Aug 2022 11:49:39 +0000 (13:49 +0200)] 
doc/source/core_plugins.rst: List buildstream-plugins

4 weeks agoMerge pull request #1748 from apache/abderrahim/storage-service
Abderrahim Kitouni [Thu, 8 Sep 2022 14:18:47 +0000 (15:18 +0100)] 
Merge pull request #1748 from apache/abderrahim/storage-service

_cas/casdprocessmanager.py: pass cert and key filenames buildbox-casd

4 weeks ago_cas/casdprocessmanager.py: pass cert and key filenames buildbox-casd abderrahim/storage-service 1748/head
Abderrahim Kitouni [Wed, 7 Sep 2022 19:55:52 +0000 (20:55 +0100)] 
_cas/casdprocessmanager.py: pass cert and key filenames buildbox-casd

It used to pass the contents of the files as command line arguments which
buildbox-casd would choke on.

6 weeks agoMerge pull request #1737 from ssssam/sam/release-workflow-fix
Tristan Van Berkom [Tue, 23 Aug 2022 15:34:58 +0000 (00:34 +0900)] 
Merge pull request #1737 from ssssam/sam/release-workflow-fix

.github/workflows/release.yml: Fix names of 'test_wheel' jobs

6 weeks agoMerge pull request #1736 from ssssam/sam/merge-workflow-fix
Tristan Van Berkom [Tue, 23 Aug 2022 15:34:10 +0000 (00:34 +0900)] 
Merge pull request #1736 from ssssam/sam/merge-workflow-fix

.github/workflows/merge.yml: Fix up job dependency after rename

6 weeks ago.github/workflows/release.yml: Fix names of 'test_wheel' jobs 1737/head
Sam Thursfield [Tue, 23 Aug 2022 15:25:08 +0000 (17:25 +0200)] 
.github/workflows/release.yml: Fix names of 'test_wheel' jobs

The 'test_wheel' job is defined twice, in ci.yml and release.yml. This
copies a small fix from ci.yml to release.yml so the jobs have useful
names.

6 weeks ago.github/workflows/merge.yml: Fix up job dependency after rename 1736/head
Sam Thursfield [Tue, 23 Aug 2022 15:18:58 +0000 (17:18 +0200)] 
.github/workflows/merge.yml: Fix up job dependency after rename

Fixes the following error in "Merge actions" workflow:

    The workflow is not valid. .github/workflows/merge.yml (Line: 46, Col: 12): Job 'publish_docs' depends on unknown job 'build'.

This is a regression introduced in https://github.com/apache/buildstream/pull/1723

6 weeks agoNEWS: Update for 1.95.2 release 1.95.2
Tristan van Berkom [Tue, 23 Aug 2022 15:08:55 +0000 (00:08 +0900)] 
NEWS: Update for 1.95.2 release

6 weeks agoMerge pull request #1723 from apache/tristan/build-wheels
Tristan Van Berkom [Tue, 23 Aug 2022 15:07:13 +0000 (00:07 +0900)] 
Merge pull request #1723 from apache/tristan/build-wheels

Build and release wheel packages

6 weeks ago.github/wheel-helpers/fetch-latest-buildbox-release.sh: Fix download link tristan/build-wheels 1723/head
Tristan van Berkom [Thu, 18 Aug 2022 05:32:44 +0000 (14:32 +0900)] 
.github/wheel-helpers/fetch-latest-buildbox-release.sh: Fix download link

Since renaming the binary tarballs to be more descriptive upstream,
we need to fix the download link where we download the binary tarballs.

See: https://gitlab.com/BuildGrid/buildbox/buildbox-integration/-/merge_requests/5

6 weeks agoSet PATH of buildbox-casd to include bundled buildbox-fuse
Sam Thursfield [Mon, 22 Aug 2022 15:38:43 +0000 (17:38 +0200)] 
Set PATH of buildbox-casd to include bundled buildbox-fuse

Otherwise, buildbox-casd falls back to its HardLinkStager backend
which does not provide sufficient capabilities to build
gnome-build-meta.

See: https://github.com/apache/buildstream/issues/1733

6 weeks agotox: Remove obsolete '-e release' mode
Sam Thursfield [Mon, 15 Aug 2022 09:27:23 +0000 (11:27 +0200)] 
tox: Remove obsolete '-e release' mode

6 weeks agodocs: Update release process
Sam Thursfield [Mon, 15 Aug 2022 09:27:14 +0000 (11:27 +0200)] 
docs: Update release process

6 weeks agoMark unstable BuildStream releases with PEP440-compatible postfix
Sam Thursfield [Fri, 12 Aug 2022 10:22:55 +0000 (12:22 +0200)] 
Mark unstable BuildStream releases with PEP440-compatible postfix

PyPI and Pip use the PEP440 version standard, which uses explicit
'.dev', '.a', '.b' and '.rc' suffixes to mark "unstable" releases.

BuildStream uses the even/odd convention. When running `pip install`
we want Pip to prefer a stable 2.0.0 release over an unstable 2.1.0
release. To make this work, all unstable releases now have a .dev0
suffix automatically added in setup.py, e.g. `2.1.0.dev0`.

This is necessary for us to fully automate releases to PyPI from GitHub
Actions.

6 weeks agoBuild and release wheels packages as part of GitHub Actions
Sam Thursfield [Tue, 16 Aug 2022 16:04:31 +0000 (18:04 +0200)] 
Build and release wheels packages as part of GitHub Actions

This updates the CI config to:

  * build wheel packages in pre-merge CI
    * download BuildBox release binaries to build the wheels
    * these can be downloaded as action artifacts
  * test built wheel packages in pre-merge CI
  * build sdist and wheel packages on each release tag, and upload
    them to Test PyPI at https://test.pypi.org/project/BuildStream/

The new workflows are based on examples at
https://cibuildwheel.readthedocs.io/en/stable/setup/, avoiding use of
the GitHub Actions from `pypa/*` as these are not permitted to be used
in apache/buildstream project.

6 weeks ago.github/workflows/merge.yml: Rename 'build' and 'publish'
Sam Thursfield [Tue, 16 Aug 2022 16:02:45 +0000 (18:02 +0200)] 
.github/workflows/merge.yml: Rename 'build' and 'publish'

We are going to build and publish more than just docs, so add _docs
to these job names to make it clear.

6 weeks agoAdd option to bundle BuildBox binaries inside Python package
Sam Thursfield [Wed, 3 Aug 2022 12:01:44 +0000 (14:01 +0200)] 
Add option to bundle BuildBox binaries inside Python package

BuildBox is not widely distributed in binary form yet. For convience,
add a mechanism to bundle prebuilt binaries in the Python wheel
packages.

Setting BST_BUNDLE_BUILDBOX=1 when setup.py runs, causes the bundled
binaries to be included in the binary package. Its up to the packager
to make appropriate binaries available in
`src/buildstream/subprojects/buildbox/`.

BuildStream will search the package subprojects/ dir when looking for
BuildBox binaries on the host in all cases, prioritizing any bundled
binaries above other ones on the host.

Related to #1712

6 weeks agoMerge pull request #1734 from apache/tristan/command-subdir-docs
Tristan Van Berkom [Tue, 23 Aug 2022 08:22:47 +0000 (17:22 +0900)] 
Merge pull request #1734 from apache/tristan/command-subdir-docs

buildelement.py: Documentation fixes and enhancements

6 weeks agobuildelement.py: Documentation fixes and enhancements 1734/head
Tristan van Berkom [Tue, 23 Aug 2022 08:05:08 +0000 (17:05 +0900)] 
buildelement.py: Documentation fixes and enhancements

 o Fix links to elements which no longer reside in BuildStream

 o Provide an example of running a srcdir != builddir build with autotools

 o Reworded some things a bit more clearly with regards to `command-subdir`,
   `conf-root` and `install-root`

6 weeks agoMerge pull request #1732 from apache/tristan/only-print-last-failure
Tristan Van Berkom [Tue, 23 Aug 2022 05:50:29 +0000 (14:50 +0900)] 
Merge pull request #1732 from apache/tristan/only-print-last-failure

_frontend/widget.py: Only save the last failure message for any given element.

6 weeks ago_frontend/widget.py: Only save the last failure message for any given element. 1732/head
Tristan van Berkom [Mon, 22 Aug 2022 06:44:27 +0000 (15:44 +0900)] 
_frontend/widget.py: Only save the last failure message for any given element.

If the element ultimately failed, it's last failure message will be printed
in the failure summary. If it previously was retried and succeeded, it will
not be printed because it will not show up in the failed tasks from
State.task_groups at session end time.

Fixes #517.

6 weeks ago_frontend/widget.py: Remove use of OrderedDict()
Tristan van Berkom [Mon, 22 Aug 2022 06:41:40 +0000 (15:41 +0900)] 
_frontend/widget.py: Remove use of OrderedDict()

Since a long time in Python 3, dict already has the ordered behavior
of OrderedDict(). No need to use OrderedDict() here.

6 weeks agoMerge pull request #1731 from apache/tristan/fix-ruamel-deprecation-warnings
Tristan Van Berkom [Sun, 21 Aug 2022 08:38:34 +0000 (17:38 +0900)] 
Merge pull request #1731 from apache/tristan/fix-ruamel-deprecation-warnings

Fix remaining ruamel deprecation warnings

6 weeks agoelement.py, _frontend/widget.py: Use _yaml.roundtrip_dump_string() 1731/head
Tristan van Berkom [Sun, 21 Aug 2022 04:37:54 +0000 (13:37 +0900)] 
element.py, _frontend/widget.py: Use _yaml.roundtrip_dump_string()

Instead of deprecated round_trip_dump() from ruamel.yaml.

This will silence a lot of the remaining pesky deprecation warnings after
running tests in tox.

This changes the output of `bst show --format '%{deps}'`, such that
the YAML output is formatted as:

  - dep-a.bst
  - dep-b.bst

Instead of [dep-a.bst, dep-b.bst], which are both valid YAML so this is
not technically an API break, however the tests/frontend/show.py test
needed to be updated.

6 weeks ago_yaml.pyx: Added roundtrip_dump_string()
Tristan van Berkom [Sun, 21 Aug 2022 04:37:10 +0000 (13:37 +0900)] 
_yaml.pyx: Added roundtrip_dump_string()

A convenience function for dumping yaml to a returned string

6 weeks agoMerge pull request #1728 from apache/tristan/fix-logging-race
Tristan Van Berkom [Sun, 21 Aug 2022 08:35:28 +0000 (17:35 +0900)] 
Merge pull request #1728 from apache/tristan/fix-logging-race

_frontend/app.py: Fixed missing messages in frontend main logging

6 weeks ago_frontend/app.py: Fixed missing messages in frontend main logging 1728/head
Tristan van Berkom [Sat, 20 Aug 2022 07:55:14 +0000 (16:55 +0900)] 
_frontend/app.py: Fixed missing messages in frontend main logging

The threaded scheduler causes the frontend message handling to be
multi-threaded, as messages are no longer passed through a message queue
but directly propagated to the frondend.

This, in conjunction with message throttling means that we need to use
a lock to protect the cached message, otherwise sometimes a message gets
cached at the same time we finish printing a message and clear the cache.

This patch also ensures that we flush the message cache more consistently
when exiting the scheduler, and causes local app.py messages to be flushed
immediately just in case.

6 weeks agoMerge pull request #1729 from apache/tristan/workspace-info-log
Tristan Van Berkom [Sat, 20 Aug 2022 08:29:35 +0000 (17:29 +0900)] 
Merge pull request #1729 from apache/tristan/workspace-info-log

_stream.py: Cleanup workspace open INFO log

6 weeks ago_stream.py: Cleanup workspace open INFO log 1729/head
Tristan van Berkom [Sat, 20 Aug 2022 08:15:45 +0000 (17:15 +0900)] 
_stream.py: Cleanup workspace open INFO log

Here we were initially printing that we're opening a workspace and then
printing that we opened it, the initial log line *can* be helpful to provide
context in case an error occurs, but let's change that to a STATUS message,
only the latter message is informative.

6 weeks agoMerge pull request #1727 from apache/tristan/quit-only-imperative-jobs
Tristan Van Berkom [Sat, 20 Aug 2022 07:53:34 +0000 (16:53 +0900)] 
Merge pull request #1727 from apache/tristan/quit-only-imperative-jobs

Continue post processing when quitting the scheduler

6 weeks ago_stream.py: Mark the imperative queues in tasks with multiple queues 1727/head
Tristan van Berkom [Fri, 19 Aug 2022 08:09:58 +0000 (17:09 +0900)] 
_stream.py: Mark the imperative queues in tasks with multiple queues

Also adjust tests to handle the new behavior

  o tests/integration/cachedfail.py: Remove xfail and workarounds for #534

  o tests/sourcecache/push.py: Fix expectation so that sources are expected
    to be pushed after a failed build

Fixes #534

7 weeks ago_scheduler: Add concept of imperative queues
Tristan van Berkom [Fri, 19 Aug 2022 08:07:55 +0000 (17:07 +0900)] 
_scheduler: Add concept of imperative queues

Impacts the behavior of quitting the scheduler (as opposed to terminating it).

If Scheduler.stop() is called, then we will stop processing jobs from
the imperative queue or any preceding queues and only continue to process
post-imperative queues.

Related to #534

7 weeks agoMerge pull request #1726 from apache/tristan/artifact-cache-refactor
Tristan Van Berkom [Thu, 18 Aug 2022 07:38:27 +0000 (16:38 +0900)] 
Merge pull request #1726 from apache/tristan/artifact-cache-refactor

Remove unused functions from ArtifactCache class

7 weeks ago_artifactcache.py: Remove unused functions pull_tree() and push_message() 1726/head
Tristan van Berkom [Thu, 18 Aug 2022 06:59:45 +0000 (15:59 +0900)] 
_artifactcache.py: Remove unused functions pull_tree() and push_message()

Fixes #781

7 weeks agotests: Refactor pull.py / push.py tests to not use deprecated methods
Tristan van Berkom [Thu, 18 Aug 2022 06:57:55 +0000 (15:57 +0900)] 
tests: Refactor pull.py / push.py tests to not use deprecated methods

As per issue #781, these functions are meant to be removed.

7 weeks agoMerge pull request #1719 from apache/tristan/shell-artifacts
Tristan Van Berkom [Wed, 17 Aug 2022 06:43:10 +0000 (15:43 +0900)] 
Merge pull request #1719 from apache/tristan/shell-artifacts

Allow artifact names in bst shell

7 weeks agotests/integration/shell.py: Test correct error messaging when abusing bst shell 1719/head
Tristan van Berkom [Tue, 16 Aug 2022 07:59:50 +0000 (16:59 +0900)] 
tests/integration/shell.py: Test correct error messaging when abusing bst shell

Ensure we get a comprehensive message when the user tries to use `bst shell`
on an artifact name for anything other than shelling into a cached build tree.

7 weeks ago_stream.py: Enhance error reporting for incorrect usage of bst shell.
Tristan van Berkom [Tue, 16 Aug 2022 07:59:21 +0000 (16:59 +0900)] 
_stream.py: Enhance error reporting for incorrect usage of bst shell.

7 weeks agotests/integration/shellbuildtrees.py: Add test for shelling into an artifact
Tristan van Berkom [Mon, 15 Aug 2022 07:58:25 +0000 (16:58 +0900)] 
tests/integration/shellbuildtrees.py: Add test for shelling into an artifact

7 weeks ago_frontend/cli.py: Rename bst shell element argument to 'target'
Tristan van Berkom [Mon, 15 Aug 2022 07:51:31 +0000 (16:51 +0900)] 
_frontend/cli.py: Rename bst shell element argument to 'target'

This will enable bash completions on artifact names for `bst shell`

7 weeks ago_stream.py: Allow loading artifacts in `bst shell`
Tristan van Berkom [Mon, 15 Aug 2022 07:49:41 +0000 (16:49 +0900)] 
_stream.py: Allow loading artifacts in `bst shell`

Now that we cache the full sandbox root for failed builds we can debug
builds directly on downloaded artifacts.

Fixes #1711

7 weeks agoMerge pull request #1722 from apache/chandan/housekeeping
Tristan Van Berkom [Wed, 17 Aug 2022 03:25:30 +0000 (12:25 +0900)] 
Merge pull request #1722 from apache/chandan/housekeeping

Minor housekeeping: Update maintainers, issue guidelines

7 weeks agoCOMMITTERS.rst: Add Abderrahim 1722/head
Chandan Singh [Tue, 16 Aug 2022 21:06:20 +0000 (22:06 +0100)] 
COMMITTERS.rst: Add Abderrahim

7 weeks agoCONTRIBUTING.rst: Remove mention of defunct issue template
Chandan Singh [Tue, 16 Aug 2022 20:58:41 +0000 (21:58 +0100)] 
CONTRIBUTING.rst: Remove mention of defunct issue template

7 weeks agoMerge pull request #1721 from apache/tristan/document-buildbox-binaries
Tristan Van Berkom [Tue, 16 Aug 2022 14:33:33 +0000 (23:33 +0900)] 
Merge pull request #1721 from apache/tristan/document-buildbox-binaries

doc/source/main_install.rst: Document where to obtain latest buildbox binaries

7 weeks agodoc/source/main_install.rst: Document where to obtain latest buildbox binaries 1721/head
Tristan van Berkom [Tue, 16 Aug 2022 08:51:28 +0000 (17:51 +0900)] 
doc/source/main_install.rst: Document where to obtain latest buildbox binaries

This effectively closes #1189

7 weeks agoMerge pull request #1718 from apache/tristan/fix-complete-artifact
Tristan Van Berkom [Mon, 15 Aug 2022 07:48:40 +0000 (16:48 +0900)] 
Merge pull request #1718 from apache/tristan/fix-complete-artifact

_context.py: Don't call utils._get_bst_api_version() at load time.

7 weeks ago_context.py: Don't call utils._get_bst_api_version() at load time. tristan/fix-complete-artifact 1718/head
Tristan van Berkom [Mon, 15 Aug 2022 06:53:07 +0000 (15:53 +0900)] 
_context.py: Don't call utils._get_bst_api_version() at load time.

Getting the version requires importing the __version__ module which not
available at bash completion time, and we need to load the buildstream.conf
file when completing artifact names (because the config may determine where
the local artifact cache resides).

This fixes stack traces which were occurring in `bst artifact checkout <TAB>`

8 weeks agoNEWS: Updating for 1.95.1 beta release 1.95.1
Tristan van Berkom [Fri, 12 Aug 2022 07:52:01 +0000 (16:52 +0900)] 
NEWS: Updating for 1.95.1 beta release

8 weeks agoMerge pull request #1715 from apache/tristan/light-cache-keys-without-sandbox-run
Tristan Van Berkom [Fri, 12 Aug 2022 07:48:46 +0000 (16:48 +0900)] 
Merge pull request #1715 from apache/tristan/light-cache-keys-without-sandbox-run

Lightweight cache keys when not running the sandbox

8 weeks agotox.ini: Fix CI for python 3.7 tristan/light-cache-keys-without-sandbox-run 1715/head
Tristan van Berkom [Fri, 12 Aug 2022 07:09:56 +0000 (16:09 +0900)] 
tox.ini: Fix CI for python 3.7

We were encountering new errors from versioneer trying to write to the
repository in order to record the version number, and so we needed to set:

  SETUPTOOLS_ENABLE_FEATURES = "legacy-editable"

8 weeks agoelement.py: Ignore environment/sandbox config when not running sandbox
Tristan van Berkom [Thu, 4 Aug 2022 06:46:24 +0000 (15:46 +0900)] 
element.py: Ignore environment/sandbox config when not running sandbox

When calculating the cache key, we can ignore sandbox related materials
when the plugin has specified BST_RUN_COMMANDS as False.

Fixes #1687

This patch also fixes cache key and completions tests which rely
on stable cache keys.

8 weeks agocompose element: setup BST_RUN_COMMANDS
Tristan van Berkom [Thu, 4 Aug 2022 06:40:28 +0000 (15:40 +0900)] 
compose element: setup BST_RUN_COMMANDS

Advertise BST_RUN_COMMANDS as False if we're not going to integrate the
dependencies.

8 weeks agosandbox.py: Clarifications to the utility of Sandbox._disable_run()
Tristan van Berkom [Thu, 4 Aug 2022 06:20:26 +0000 (15:20 +0900)] 
sandbox.py: Clarifications to the utility of Sandbox._disable_run()

In the past, BuildStream had internal sandbox implementations which
would require staging files on the local filesystem in order to run,
but now we use buildbox-run exclusively and always use CasBasedDirectory.

The _use_cas_based_directory() method was removed in the commit
b487a11f6ee529c9cba505106d054c92c16e864c, and this was actually the
only reason why we had the Element.BST_RUN_COMMANDS flag at the time.

We still want BST_RUN_COMMANDS for other reasons, and it is useful
to keep the Sandbox._disable_run() function, but it's only purpose
now is to ensure the invariant that plugins who set BST_RUN_COMMANDS
to False do not actually run commands.

8 weeks agoMerge pull request #1707 from apache/tristan/full-build-tree
Tristan Van Berkom [Thu, 11 Aug 2022 13:54:10 +0000 (22:54 +0900)] 
Merge pull request #1707 from apache/tristan/full-build-tree

Cache full build tree for debugging failed builds

8 weeks agotests/integration/shellbuildtrees.py: Test build shells with compose elements tristan/full-build-tree 1707/head
Tristan van Berkom [Mon, 1 Aug 2022 07:30:55 +0000 (16:30 +0900)] 
tests/integration/shellbuildtrees.py: Test build shells with compose elements

Test cases where the compose element is successful and also in cases where
the integration commands have failed.

Don't bother testing this with downloading from a remote cache, that is
sufficiently tested.

8 weeks agotests/integration/shellbuildtrees.py: Test build shell with script element
Tristan van Berkom [Mon, 1 Aug 2022 07:39:05 +0000 (16:39 +0900)] 
tests/integration/shellbuildtrees.py: Test build shell with script element

8 weeks agoelement.py, sandbox.py: Removing usebuildtree codepath from staging
Tristan van Berkom [Sun, 31 Jul 2022 09:43:31 +0000 (18:43 +0900)] 
element.py, sandbox.py: Removing usebuildtree codepath from staging

This also cleans up a mess where Element was inappropriately tagging the
Sandbox object with 'usebuildtree' state.

8 weeks agoCache the sandbox root when caching build trees
Tristan van Berkom [Sun, 31 Jul 2022 07:32:39 +0000 (16:32 +0900)] 
Cache the sandbox root when caching build trees

  o _artifact.py: Load and store buildroot, added get_buildroot() API

  o _artifactcache.py: Push and pull buildroot

  o _stream.py: Don't require dependencies when using `--use-buildtree`, the
    dependencies are no longer needed.

    Also remove unused function _buildtree_pull_required()

  o element.py: Use the artifact buildroot to implement shells which use
    the buildtree

  o tests/integration/shellbuildtrees.py: Updated test expectations, and
    also now test that `--use-builtree` works without pulling dependencies

8 weeks ago_artifact.py, element.py, buildqueue.py: Don't try to calculate artifact size
Tristan van Berkom [Sun, 31 Jul 2022 07:18:22 +0000 (16:18 +0900)] 
_artifact.py, element.py, buildqueue.py: Don't try to calculate artifact size

The artifact caching codepath was trying to calculate artifact size, in
a way that it is impossible to accurately calculate. Furthermore the
returned size is propagated through some scheduler codepaths and then
simply ignored.

Lets drop this deadcode.

8 weeks ago_versions.py, _artifact.py: Bumping artifact versions
Tristan van Berkom [Sun, 31 Jul 2022 07:04:19 +0000 (16:04 +0900)] 
_versions.py, _artifact.py: Bumping artifact versions

Adjusting artifact versioning for the purpose of adding the new buildroot field

  o _versions.py: Bumping overall artifact version, breaking cache keys

  o _artifact.py: Bumping artifact protobuf version because of new field

This also updates the cachekey test and completions test which assume
artifact cache key stability.

8 weeks ago_protos/buildstream/v2/artifact.proto: Adding buildroot to the artifact proto
Tristan van Berkom [Sun, 31 Jul 2022 07:03:33 +0000 (16:03 +0900)] 
_protos/buildstream/v2/artifact.proto: Adding buildroot to the artifact proto

And regenerating the grpc code for the new artifact field.

8 weeks agoelement.py: Remove redundant context from Plugin.warn() invokation
Tristan van Berkom [Sun, 31 Jul 2022 06:17:05 +0000 (15:17 +0900)] 
element.py: Remove redundant context from Plugin.warn() invokation

When calling `self.warn()` there is no need to print the element name or
print "WARNING", this context is implied by calling `Plugin.warn()` already.

8 weeks agoMerge pull request #1710 from jjardon/jjardon/lzip
Tristan Van Berkom [Mon, 8 Aug 2022 09:57:16 +0000 (18:57 +0900)] 
Merge pull request #1710 from jjardon/jjardon/lzip

doc/source/main_install.rst: Buildstream depends on lzip

8 weeks agoUpdate main_install.rst 1710/head
Tristan Van Berkom [Mon, 8 Aug 2022 09:56:47 +0000 (18:56 +0900)] 
Update main_install.rst

2 months agodoc/source/main_install.rst: Buildstream depends on lzip
Javier Jardón [Tue, 2 Aug 2022 10:42:16 +0000 (11:42 +0100)] 
doc/source/main_install.rst: Buildstream depends on lzip

This is a dependency of the core tar plugin

2 months agoMerge pull request #1705 from jjardon/jjardon/bst2_install
Jürg Billeter [Thu, 28 Jul 2022 14:29:53 +0000 (16:29 +0200)] 
Merge pull request #1705 from jjardon/jjardon/bst2_install

doc/source/main_install.rst: Buildstream2 doesnt have default plugins anymore

2 months agoMerge pull request #1704 from apache/tristan/source-set-ref-mapping-node
Tristan Van Berkom [Thu, 28 Jul 2022 09:28:57 +0000 (18:28 +0900)] 
Merge pull request #1704 from apache/tristan/source-set-ref-mapping-node

source.py: Pass MappingNode to Source.set_ref()

2 months agoplugin.py,source.py,types.py: Documenting SourceRef more thoroughly tristan/source-set-ref-mapping-node 1704/head
Tristan van Berkom [Thu, 28 Jul 2022 08:10:49 +0000 (17:10 +0900)] 
plugin.py,source.py,types.py: Documenting SourceRef more thoroughly

This patch adds some cross references to related docstrings and adds
a section to the Source documentation explaining how source implementations
are intended to handle the SourceRef.

2 months agoelement.py: Provide some additional links in docstrings
Tristan van Berkom [Thu, 28 Jul 2022 08:10:04 +0000 (17:10 +0900)] 
element.py: Provide some additional links in docstrings

2 months ago__init__.py: Import SourceRef
Tristan van Berkom [Thu, 28 Jul 2022 08:09:03 +0000 (17:09 +0900)] 
__init__.py: Import SourceRef

Can be useful for plugin implementors using strong typing, should
be exported at the toplevel of the `buildstream` module like all other
public symbols.

2 months agodoc/source/main_install.rst: Buildstream2 doesnt have default plugins anymore 1705/head
Javier Jardón [Tue, 26 Jul 2022 11:28:01 +0000 (12:28 +0100)] 
doc/source/main_install.rst: Buildstream2 doesnt have default plugins anymore

2 months agosource.py: Pass MappingNode to Source.set_ref()
Tristan van Berkom [Tue, 26 Jul 2022 08:44:59 +0000 (17:44 +0900)] 
source.py: Pass MappingNode to Source.set_ref()

The API documentation calls for a MappingNode so we should be passing
the plugin a MappingNode here.

Fixes #1685

2 months agotests/internals/yaml.py: Test behaviors of None types in MappingNode
Tristan van Berkom [Wed, 27 Jul 2022 06:52:06 +0000 (15:52 +0900)] 
tests/internals/yaml.py: Test behaviors of None types in MappingNode

2 months agonode.pyx, _yaml.pyx: Support setting None values on MappingNode objects
Tristan van Berkom [Wed, 27 Jul 2022 06:46:30 +0000 (15:46 +0900)] 
node.pyx, _yaml.pyx: Support setting None values on MappingNode objects

This patch adds support for:
  * Setting none values on Node structures, e.g. foo["bar"] = None
  * Saving Node structures which contain None values to YAML

When saving None values to YAML, we save them as empty strings. This is
a valid None value by YAML specifications, but will later be loaded by
BuildStream as an empty string.

This change is required in order to address #1685, which requires the
ability to set None values in MappingNode objects.

2 months agoMerge pull request #1702 from apache/tristan/print-error-messages-before-prompt
Tristan Van Berkom [Wed, 27 Jul 2022 05:28:38 +0000 (14:28 +0900)] 
Merge pull request #1702 from apache/tristan/print-error-messages-before-prompt

Print error messages before prompt

2 months agotests/integration/interactive_build.py: Use integration cache for sources
Tristan van Berkom [Tue, 26 Jul 2022 08:41:24 +0000 (17:41 +0900)] 
tests/integration/interactive_build.py: Use integration cache for sources

These tests don't need to redownload the alpine image for each test.

2 months ago_frontend/app.py: Flush cached messages when handling a failure. tristan/print-error-messages-before-prompt 1702/head
Tristan van Berkom [Tue, 26 Jul 2022 06:18:58 +0000 (15:18 +0900)] 
_frontend/app.py: Flush cached messages when handling a failure.

It's very confusing to get an interactive prompt and not see the detailed
failure message on screen - make sure we print the pending messages when
handling a failure.

2 months ago_message.py: Add type annotations
Tristan van Berkom [Tue, 26 Jul 2022 06:13:36 +0000 (15:13 +0900)] 
_message.py: Add type annotations

2 months ago_messenger.py: Document some undocumented symbols, and correct one type
Tristan van Berkom [Tue, 26 Jul 2022 06:01:33 +0000 (15:01 +0900)] 
_messenger.py: Document some undocumented symbols, and correct one type

The element_key passed around in Message objects is a _DisplayKey, not str.

2 months agoMerge pull request #1703 from apache/juerg/zip-strict
Jürg Billeter [Tue, 26 Jul 2022 08:28:07 +0000 (10:28 +0200)] 
Merge pull request #1703 from apache/juerg/zip-strict

Do not use `zip(..., strict=True)` as it requires Python 3.10

2 months agoDo not use `zip(..., strict=True)` as it requires Python 3.10 juerg/zip-strict 1703/head
Jürg Billeter [Tue, 26 Jul 2022 07:36:35 +0000 (09:36 +0200)] 
Do not use `zip(..., strict=True)` as it requires Python 3.10

2 months agoMerge pull request #1697 from apache/juerg/artifact-cache-opt
Jürg Billeter [Tue, 26 Jul 2022 05:30:56 +0000 (07:30 +0200)] 
Merge pull request #1697 from apache/juerg/artifact-cache-opt

_artifact.py: cache(): Optimize capture of metadata files

2 months agoMerge pull request #1698 from apache/juerg/directory-import-opt
Jürg Billeter [Tue, 26 Jul 2022 05:30:36 +0000 (07:30 +0200)] 
Merge pull request #1698 from apache/juerg/directory-import-opt

storage: Add `collect_result` parameter to `import_files()`

2 months agoMerge pull request #1700 from apache/juerg/stage-access
Jürg Billeter [Tue, 26 Jul 2022 05:30:19 +0000 (07:30 +0200)] 
Merge pull request #1700 from apache/juerg/stage-access

Specify access credentials in `stage_directory()`

2 months agoMerge pull request #1701 from apache/tristan/fix-load-cross-project-element
Tristan Van Berkom [Tue, 26 Jul 2022 05:28:46 +0000 (14:28 +0900)] 
Merge pull request #1701 from apache/tristan/fix-load-cross-project-element

Fix loading cross project element

2 months agotests/plugins/loading.py: Check loading of subproject elements via command line tristan/fix-load-cross-project-element 1701/head
Tristan van Berkom [Mon, 25 Jul 2022 15:40:19 +0000 (00:40 +0900)] 
tests/plugins/loading.py: Check loading of subproject elements via command line

Since this test case is a tricky load pattern, ensure that our changes which
ensures the toplevel project gets loaded first when addressing subproject
elements on the command line does not break this project.

2 months agotests/format/junctions.py: Test complex cross junction include scenario
Tristan van Berkom [Mon, 25 Jul 2022 07:15:06 +0000 (16:15 +0900)] 
tests/format/junctions.py: Test complex cross junction include scenario

In this scenario, loading a file directly across a junction, without using
a depending toplevel project element, produces stack traces.

2 months ago_loader/loader.py: Fix a couple of issues with loading cross junction elements
Tristan van Berkom [Mon, 25 Jul 2022 14:53:26 +0000 (23:53 +0900)] 
_loader/loader.py: Fix a couple of issues with loading cross junction elements

  * Ensure the toplevel project is resolved first when addressing subproject
    elements on the command line

  * Ignore empty provenances when detecting circular references

Fixes #1455

2 months ago_project.py: Remove deadcode Project initializer `load_project`
Tristan van Berkom [Mon, 25 Jul 2022 09:15:27 +0000 (18:15 +0900)] 
_project.py: Remove deadcode Project initializer `load_project`

This is an artifact from earlier days and is now meaningless.

2 months agoMerge pull request #1699 from apache/tristan/fix-potential-race
Tristan Van Berkom [Mon, 25 Jul 2022 07:13:01 +0000 (16:13 +0900)] 
Merge pull request #1699 from apache/tristan/fix-potential-race

element.py: Reimplementation of retrying failed builds in non-strict mode

2 months agoelement.py: Lower expectations when discarding failed builds in non-strict mode tristan/fix-potential-race 1699/head
Tristan van Berkom [Mon, 25 Jul 2022 05:43:38 +0000 (14:43 +0900)] 
element.py: Lower expectations when discarding failed builds in non-strict mode

We were trying to compare the potential strong key against the artifact
strong key, but the potential strong key is not guaranteed to be known when
loading the artifact.

For now, we just retry failed builds in non-strict mode as long as the failed
build artifact key is not equal to the element's strict key, as we know that
dependencies *might* have changed since the failed build occurred.

2 months agoMerge pull request #1696 from apache/tristan/fix-cross-junction-plugins
Tristan Van Berkom [Mon, 25 Jul 2022 05:53:32 +0000 (14:53 +0900)] 
Merge pull request #1696 from apache/tristan/fix-cross-junction-plugins

Avoid prematurely resolving projects when loading plugins across junctions

2 months agotests/plugin/loading.py: Regression test for complex cross junction scenario tristan/fix-cross-junction-plugins 1696/head
Tristan van Berkom [Fri, 22 Jul 2022 09:46:18 +0000 (18:46 +0900)] 
tests/plugin/loading.py: Regression test for complex cross junction scenario

This sets up a complex test scenario which provokes the junction
plugin origin to prematurely resolve projects, triggering issues like #1686.

2 months ago_pluginfactory/pluginoriginjunction.py: Don't force load projects
Tristan van Berkom [Fri, 22 Jul 2022 09:44:49 +0000 (18:44 +0900)] 
_pluginfactory/pluginoriginjunction.py: Don't force load projects

Cross junction plugin loading must not require completely resolving projects,
as this will impede the functionality of cross junction includes.

Fixes #1686