* Add OIDC tokens to gitlab-ci jobs
This should allow us to start issuing just-in-time generated
credentials for CI jobs that need to modify binary mirrors. The "aud"
claim of the token describes what the token is allowed to do. The
claim is verified against a set of rules on the IAM role using signed
information from GitLab. See spack-infrastructure for the claim
verification logic.
---------
Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
This commit replaces the internal representation of deptypes with `int`, which is more compact
and faster to operate with.
Double loops like:
```
any(x in ys for x in xs)
```
are replaced by constant operations bool(xs & ys), where xs and ys are dependency types.
Global constants are exposed for convenience in `spack.deptypes`
Currently the configuration of the pipeline is such that
there are multiple "optimal" solutions. This is due to
the pipeline making ~lld the default for LLVM, but leaving
+libomptarget from package.py
Since LLVM recipe has a:
conflicts("~lld", "+libomptarget")
clingo is forced to change one of the two defaults, and
the penalty for doing so is the same for each. Hence, it
is random whether we'll get +libomptarget+lld
or ~libomptarget~lld.
This fixes it by changing also the default for libomptarget.
Instead of pointing to the image on DockerHub, which rate limits us and
causes pipeline failures durying busy times, use the version at ghcr.
And we might as well use the ghcr version everywhere else too.
Smart alias completion introduced in #39499 wasn't as smart as it needed to be, and
would complete any invalid command prefix and some env names with alias names.
- [x] don't complete aliases if there are no potential completions
e.g., don't convert `spack isnotacommand` -> `spack concretize`
- [x] don't complete with an aliases if we're not looking at a top-level subcommand.
* Perform external spec detection with multiple workers
The logic to perform external spec detection has been refactored
into classes. These classes use the GoF "template" pattern to account
for the small differences between searching for "executables" and
for "libraries", while unifying the larger part of the algorithm.
A ProcessPoolExecutor is used to parallelize the work.
* Speed-up external find by tagging detectable packages automatically
Querying packages by tag is much faster than inspecting the repository,
since tags are cached. This commit adds a "detectable" tag to every
package that implements the detection protocol, and external detection
uses it to search for packages.
* Pass package names instead of package classes to workers
The slowest part of the search is importing the Python modules
associated with candidate packages. The import is done serially
before we distribute the work to the pool of executors.
This commit pushes the import of the Python module to the job
performed by the workers, and passes just the name of the packages
to the executors.
In this way imports can be done in parallel.
* Rework unit-tests for Windows
Some unit tests were doing a full e2e run of a command
just to check a input handling. Make the test more
focused by just stressing a specific function.
Mark as xfailed 2 tests on Windows, that will be fixed
by a PR in the queue. The tests are failing because we
monkeypatch internals in the parent process, but the
monkeypatching is not done in the "spawned" child
process.
* Write timing information for installs from cache
* CI: aggregate and upload install_times.json to artifacts
* CI: Don't change root directory for artifact generation
* Flat event based timer variation
Event based timer allows for easily starting and stopping timers without
wiping sub-timer data. It also requires less branching logic when
tracking time.
The json output is non-hierarchical in this version and hierarchy is
less rigidly enforced between starting and stopping.
* Add and write timers for top level install
* Update completion
* remove unused subtimer api
* Fix unit tests
* Suppress timing summary option
* Save timers summaries to user_data artifacts
* Remove completion from fish
* Move spack python to script section
* Write timer correctly for non-cache installs
* Re-add hash to timer file
* Fish completion updates
* Fix null timer yield value
* fix type hints
* Remove timer-summary-file option
* Add "." in front of non-package timer name
---------
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
These commands are currently broken on powershell (Windows) due to
improper use of the InvokeCommand commandlet and a lack of direct
support for the `--pwsh` argument in `spack load`, `spack unload`,
and `spack env deactivate`.
Setting the undocumented variable SPACK_CONCRETIZER_REQUIRE_CHECKSUM
now causes the solver to avoid accounting for versions that are not checksummed.
This feature is used in CI to avoid spurious concretization against e.g. develop branches.
`compgen -W` does not behave the same way in zsh as it does in bash; it seems not to
actually generate the completions we want.
- [x] add a zsh equivalent and `_compgen_w` to abstract it away
- [x] use `_compgen_w` instead of `compgen -W`
Bash completion is now smarter about handling aliases. In particular, if all completions
for some input command are aliased to the same thing, we'll just complete with that thing.
If you've already *typed* the full alias for a command, we'll complete the alias.
So, for example, here there's more than one real command involved, so all aliases are
shown:
```console
$ spack con
concretise concretize config containerise containerize
```
Here, there are two possibilities: `concretise` and `concretize`, but both map to
`concretize` so we just complete that:
```console
$ spack conc
concretize
```
And here, the user has already typed `concretis`, so we just go with it as there is only
one option:
```console
spack concretis
concretise
```
From a user:
> Aargh.
> ```
> ==> Error: concretise is not a recognized Spack command or extension command; check with `spack commands`.
> ```
To make things easier for our friends in the UK, this adds `concretise` and
`containerise` aliases for the `spack concretize` and `spack containerize` commands.
- [x] add aliases
- [x] update completions
* VTK: Add patch for python 3.8 support
* CI: Re-enable VisIt in CI
* Configure spec matrix for stack with VisIt
* Add pugixml dep for 8.2.0
* Make VTK and ParaView consistent on proj dep
* OpenMPI 3: provides MP support by default
* Add details on proj dep in ParaView
* Add python 3.8 to test mock repo
* Patches to get VisIt VTK interface
* CI: Disable VisIt with GUI in DAV
* AMReX: 23.06+ Multi-Dim Support
This updated the Spack package to allow to install AMReX, modules of
AMReX in E4S deployments and dependent packages with support for
multiple dimensions. Due to an upstream change in AMReX, we do not
longer need to ship three, binary incompatible package variants.
* [E4S] oneAPI AMReX < 23.06 Variant
Work-around the auto-concretization to the multi-dim of `dimensions`,
which only in 23.06+ became a multi-variant.
* e4s cray rhel ci: temporarily disable amrex build until spurious ci failure can be resolved
---------
Co-authored-by: eugeneswalker <eugenesunsetwalker@gmail.com>
Refactor gitlab ci configs so that mac and cray jobs can reuse as much higher level
configuration as possible.
* CI: remove redundant sections
* CI: Include base linux CI configs in cray stacks
Relocation and runner mapping is consistent between cray and linux runners.
* Export user cache path in before script
* CI: add GPG root for mac runners
* Disable user configs
Metal runners share a ~ directory
* Disable user config and add configs in activate env
* CI: Refactor ci reproducer
* Autostart container
* Reproducer paths match CI paths
* Generate start scripts for docker and reproducer
* CI: Add interactive and gpg options to reproduce-build
* Interactive will determine if the docker container persists
after running reproduction.
* GPG path/url allow downloading GPG keys needed for binary
cache download validation. This is important for running
reproducer for protected CI jobs.
* Add exit_on_failure option to CI scripts
* CI: Add runtime option for reproducer
`spack buildcache list` did not have a way to display the namespace of
packages in the buildcache. This PR adds that functionality.
For consistency's sake, it moves the `-N/--namespace` arg definition to
the `common/arguments.py` and modifies `find`, `solve`, `spec` to use
the common definition.
Previously, `find` was using `--namespace` (singular) to control whether
to display the namespace (it doesn't restrict the search to that
namespace). The other commands were using `--namespaces` (plural). For
backwards compatibility and for consistency with `--deps`, `--tags`,
etc, the plural `--namespaces` was chosen. The argument parser ensures
that `find --namespace` will continue to behave as before.
Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
* Add rewrite of spack checksum to include --verify and better add versions to package.py files
* Fix formatting and remove unused import
* Update checksum unit-tests to correctly test multiple versions and add to package
* Remove references to latest in stage.py
* Update bash-completion scripts to fix unit tests failures
* Fix docs generation
* Remove unused url_dict argument from methods
* Reduce chance of redundant remote_versions work
* Add print() before tty.die() to increase error readablity
* Update version regular expression to allow for multi-line versions
* Add a few unit tests to improve test coverage
* Update command completion
* Add type hints to added functions and fix a few py-lint suggestions
* Add @no_type_check to prevent mypy from failing on pkg.versions
* Add type hints to format.py and fix unit test
* Black format lib/spack/spack/package_base.py
* Attempt ignoring type errors
* Add optional dict type hint and declare versions in PackageBase
* Refactor util/format.py to allow for url_dict as an optional parameter
* Directly reference PackageBase class instead of using TypeVar
* Fix comment typo
---------
Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov>
* commands: provide more information to Command
* fish: Add script to generate fish completion
* fish: auto prepend `spack` command to avoid duplication
* fish: impove completion generation code readability
* commands: replace match-case with if-else
* fish: fix optspec variable name prefix
* fish: fix return value in get_optspecs
* fish: fix return value in get_optspecs
* format: split long line and trim trailing space
* bugfix: replace f-string with interpolation
* fish: compete more specs and some fixes
* fish: complete hash spec starts with /
* fish: improve compatibility
* style: trim trailing whitespace
* commands: add fish to update args and update tests
* commands: add fish completion file
* style: merge imports
* fish: source completion in setup-env
* fish: caret only completes dependencies
* fish: make sure we always get same order of output
* fish: spack activate
only show installed packages that have extensions
* fish: update completion file
* fish: make dict keys sorted
* Blacken code
* Fix bad merge
* Undo style changes to setup-env.fish
* Fix unit tests
* Style fix
* Compatible with fish_indent
* Use list for stability of order
* Sort one more place
* Sort more things
* Sorting unneeded
* Unsort
* Print difference
* Style fix
* Help messages need quotes
* Arguments to -a must be quoted
* Update types
* Update types
* Update types
* Add type hints
* Change order of positionals
* Always expand help
* Remove shared base class
* Fix type hints
* Remove platform-specific choices
* First line of help only
* Remove unused maps
* Remove suppress
* Remove debugging comments
* Better quoting
* Fish completions have no double dash
* Remove test for deleted class
* Fix grammar in header file
* Use single quotes in most places
* Better support for remainder nargs
* No magic strings
* * and + can also complete multiple
* lower case, no period
---------
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Use curly braces instead of quotes to enclose value or text in Tcl
modulefile. Within curly braces Tcl special characters like [, ] or $
are treated verbatim whereas they are evaluated within quotes.
Curly braces is Tcl recommended way to enclose verbatim content [1].
Note: if curly braces charaters are used within content, they must be
balanced. This point has been checked against current repository and no
unbalanced curly braces has been spotted.
Fixes#24243
[1] https://wiki.tcl-lang.org/page/Tcl+Minimal+Escaping+Style
Scott Wittenburg