Currently, the DOWNLOAD macro is context-dependent and expects
the PKG variable to be set to the current package.
This is not so nice.
Change the macro to expect the upper-case package name as a
parameter, rather than expect it from a variable.
Adapt the caller accordingly.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The FLOCK variable is context-dependent, and expects the PKG
variable to be set to the current package.
This is not so nice. Besides, it is used in a single location.
Get rid of this intermediate variable, and directly use flock
where we need it.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Currently, that variable is context-dependent, as it expects the PKG
variable to exist and be defined to the current package.
This is not so clean, so change the variable to a callable macro.
Adapt the caller accordingly.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Currently, the internal DOWNLOAD_URIS variable is set to be a list of
options to pass to the download wrapper, with each URI passed as
'-u URI'.
This precludes using that variable to get just the list of URIs, in
case we need to do something else with it.
Fix the variable to really only contain the list of URIs.
Adapt the caller accordingly.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The ssh command was added back in 2011 with commit c61788f09 (GENTARGETS:
add support for scp://) and was used to check that the remote file
existed, back when we supported 'make source-check'.
However, in 2017, with commit bf28a165d (pkg-{download, generic}: remove
source-check), we actually removed support for source-check.
The SSH command however was not removed then, and stuck, even though
nothing ever uses it It is not even exported in the environment, and scp
does not use it either (it has -S to specify an ssh-compatible program).
Get rid of it.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When $(TOPDIR)/dl is a symlink, checking out git submodules can fail,
as reported by Michael in #11086.
To reproduce a similarly-related mis-behaviour:
$ mkdir -p foo/bar foo/buz
$ cd foo/bar
$ ln -s ../buz meh
$ cd meh
$ cd ../../foo
The last command should not succeed, because, relative to meh, there is
no ../../foo directory; we would expect it to be ../../../foo, instead.
But since meh is a symlink to a directory, then a relative path from that
symlink is interpreted as relative to the derefrenced directory, i.e.
from buz in this case.
But where this gets even weirder, is that, if the last command is
replaced by:
$ cd ../../../foo
then it still works, too.
And that is the root of Michael's issue: the dl directory in Buildroot's
TOPDIR is a symlink to a similarly-named directory one directory higher,
which then confuses relative paths, which gets especially and noticeably
bad for git submodules.
Avoid this strangeness, and just use so-called "physical" path, i.e. a
path where all symlinks to directories have been dereferenced.
Fixes: #11086
Reported-by: Michael Nosthoff <posted@heine.so>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Michael Nosthoff <posted@heine.so>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Now that packges may have hashes for their downloaded files in a
versioned subdir or in the unversioned hash file, we can no longer
hard-code the path to the package hash file anymore.
Instead, we now make use of the per-package variable, that points to the
package hash file.
Note: of the packages for which we offer a version choice, almost none,
but some of the qt5 ones, have a per-version hash file, so we still use
the unversioned hash file for them. As for the few qt5 packages that do
have a per-version hash file for their licensing terms, they've already
been updated to duplicate their download hashes in both the unversioned
and per-version hash files. So, one way or the other, no hash check
would go missing with this change.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Baruch Siach <baruch@tkos.co.il>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The download wrapper call is currently always being displayed, even
without V=1, which is a bit annoying. It shows something like this:
thomas@windsurf:~/projets/buildroot (master)$ make tslib-source
>>> tslib 1.16 Downloading
PATH="/home/thomas/projets/buildroot/output/host/bin:/home/thomas/projets/buildroot/output/host/sbin:/usr/local/bin:/usr/bin:/bin:/home/thomas/.rvm/bin:/usr/local/sbin:/usr/sbin:/home/thomas/.rvm/bin:/home/thomas/sys/bin:/home/thomas/.gem/ruby/2.1.0/bin:/home/thomas/.rvm/bin" BR2_DL_DIR=/home/thomas/dl BUILD_DIR=/home/thomas/projets/buildroot/output/build O=/home/thomas/projets/buildroot/output flock /home/thomas/dl/tslib/ support/download/dl-wrapper -c '1.16' -d '/home/thomas/dl/tslib' -D '/home/thomas/dl' -f 'tslib-1.16.tar.xz' -H 'package/tslib//tslib.hash' -n 'tslib-1.16' -N 'tslib' -o '/home/thomas/dl/tslib/tslib-1.16.tar.xz' -u https+https://github.com/kergoth/tslib/releases/download/1.16 -u http\|urlencode+http://sources.buildroot.net/tslib -u http\|urlencode+http://sources.buildroot.net --
Let's silence this dl-wrapper call by prepending with $(Q).
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The site method only ever applies to the main download, while extra
downloads are always to be fetched with wget.
However, the site method is prepended to the URL from within the
DOWNLOAD macro (well, a variable evaluated in the DOWNLOAD macro),
which is called for each download of a package, thus effectively
prepending the site method to all downloads, even the extra ones (and
the patches).
We fix that by prepending the site method from within the
generic-package infra, so that it only applies to the main download.
For that, we move the main _SOURCE out of the foreach loop, so that
we can prepend the site-method to it, without impacting the other
downloads.
Reported-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, when the main download if from an SCM, we exclude all
downloads from being hash-checked, on the assumption that we don't
have hashes for downloads from an SCM.
However, the exclusion is computed on the DOWNLOAD macro, which is
called for each download of a package, thus effectively disabling
hash checks for extra downloads, even though those are only ever
download with wget.
What we really wanted to do, in fact, was to exclude just the main
download.
We fix that by appending the main source file to the global list of
excluded files, from within the generic-package infra itself.
Reported-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Fixes
http://autobuild.buildroot.net/results/734/7342b25f01d60cafc4a5422a5a1c282629f361c5/
$(dir does not strip last path separator and the wget wrapper adds a '/'
between the -u argument and the filename when it reconstructs the URL,
leading us to have two slashes before the filename when accessing the
upstream location.
Most upstream servers ignores this, but not all - Leading to download issues
from pypi.
Notice: As pointed out by Arnout, we cannot simply use $(PKG)_SITE here as
_PATCH and _EXTRA_DOWNLOADS may contain full URLs.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
For existing setups, the global donload directory may have a lot of the
required archives, so look into there before attempting a download.
We simply hard-link them if found there and not in the new per-package
loaction. Then we resume the existing procedure (which means the new
hardlink will get removed if it happened to not match the hash).
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The infrastructure needs to give the 'dl_dir' to the dl-wrapper which in its
turn needs to give it to the helper. It will only be used by the 'git'
helper as of now.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
In order to introduce the cache mechanisms, we need to have a lock on the
$(LIBFOO_DL_DIR), otherwise it would be impossible to do parallel download
(a shared DL_DIR for two buildroot instances) without risking conflicts.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Since we introduced subdirectories to the DL_DIR, we need to support them
in the PRIMARY and BACKUP mirrors as they evolve to the new tree
structure.
We check first the new URI (with the subdir), and in case of failure, we
check without. By checking both URIs, we ensure that old mirror are usable.
Also, add a missing qstrip call for BR2_BACKUP_SITE.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
At the moment, it means that we make sure that BR2_DL_DIR is created, in
the future, it will make sure that BR2_DL_DIR/PKG_NAME/ is created.
[Peter: drop trailing / on mkdir]
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Let the infrastructure use the already existing variable $(PKG)_DL_DIR
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Some arguments can be empty, so we need to add single quote around them
But since they are sinple quote; double quote inside them are now not
interpreted by the shell and as such must be removed; hence the use of
qstrip.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The goal here is to simplify the infrastructure by putting most of the
code in the dl-wrapper as it is easier to implement and to read.
Most of the functions were common already, this patch finalizes it by
making the pkg-download.mk pass all the parameters needed to the
dl-wrapper which in turn will pass everything to every backend.
The backend will then cherry-pick what it needs from these arguments
and act accordingly.
It eases the transition to the addition of a sub directory per package
in the DL_DIR, and later on, a git cache.
[Peter: drop ';' in BR_NO_CHECK_HASH_FOR in DOWNLOAD macro and swap cd/rm
-rf as mentioned by Yann, fix typos]
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently all download helpers accepts the local output file, the remote
locations, the changesets and so on... as positional arguments.
This was well and nice when that's was all we needed.
But then we added an option to quiesce their verbosity, and that was
shoehorned with a trivial getopts, still keeping all the existing
positional arguments as... positional arguments.
Adding yet more options while keeping positional arguments will not be
very easy, even if we do not envision any new option in the foreseeable
future (but 640K ought to be enough for everyone, remember? ;-) ).
Change all helpers to accept a set of generic options (-q for quiet and
-o for the output file) as well as helper-specific options (like -r for
the repository, -c for a changeset...).
Maxime:
Changed -R to -r for recurse (only for the git backend)
Changed -r to -u for URI (for all backend)
Change -R to -c for cset (for CVS and SVN backend)
Add the export of the BR_BACKEND_DL_GETOPTS so all the backend wrapper
can use the same option easily
Now all the backends use the same common options.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
In current Buildroot, clashes occur between the variables _NAME and
_BASE_NAME for two packages called foo and foo-base, i.e.
Package foo:
FOO_NAME = foo
FOO_BASE_NAME = foo-1.2.3
Package foo-base:
FOO_BASE_NAME = foo-base
FOO_BASE_BASE_NAME = foo-base-4.5.6
where variable FOO_BASE_NAME is clashing between these two packages.
Specific cases where this clash is already existing are:
- alljoyn-base
- alljoyn-tcl-base
- perl-xml-sax-base
The problem is generic and can occur for a number of variables in Buildroot.
A non-exhaustive list:
<pkg>_BASE and <pkg>_BASE_NAME
<pkg>_BASE_NAME and <pkg>_RAW_BASE_NAME
<pkg>_DIR and <pkg>_DL_DIR
<pkg>_VERSION and <pkg>_DL_VERSION
<pkg>_SOURCE and <pkg>_TARGET_SOURCE
<pkg>_INSTALL_IMAGES and <pkg>_TARGET_INSTALL_IMAGES (same for _STAGING and _TARGET)
<pkg>_LICENSE_FILES and <pkg>_MANIFEST_LICENSE_FILES
<pkg>_DEPENDENCIES and <pkg>_FINAL_DEPENDENCIES
One solution is to use another separator than '_' to separate the
package name from the rest of the variable name. For example, a double
underscore:
FOO__NAME
FOO__BASE_NAME
FOO_BASE__NAME
FOO_BASE__BASE_NAME
However, making that change for only this case means that the variable
naming is no longer consistent. And making the change for all variables has
a large impact, also on certain user scripts.
For now, keep it simple, and rename FOO_BASE_NAME into FOO_BASENAME, so that
the variables become:
FOO_NAME
FOO_BASENAME
FOO_BASE_NAME
FOO_BASE_BASENAME
For consistency, also adapt FOO_RAW_BASE_NAME. Since FOO_RAW_BASENAME would
still pose a conflict with a package called 'foo-raw', take the opportunity
to rename it into FOO_BASENAME_RAW instead, which does not pose a conflict
as we have no variable called FOO_RAW.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Sam Voss <sam.voss@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This feature is not used by anyone in the core developpers and makes a
drastic simplification of the pkg-download infrastructure harder.
The future patch will move much of what's in the current pkg-download.mk
file into the dl-wrapper which is a shell script.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
This reverts commit 1e5a8916b2.
The idea was that the version string can be derived because we know the
package name.
However, this patch does not account for the fact that $(pkgname) always
points to the latest pacakge scanned, which in all other situation we're
using it, is the current package, because it is used inside one ot he
xxx-inner macros that are $(eval)ualed. So $(pkgname) is evaluated
"early" and gets the expected value.
However, the github value is not in one of those macros, so it gets
evaluated "late", when doing the actual download. So, by that time,
$(pkgname) will expand to the last package scanned, which is actuall the
manual (without a br2-external tree).
That would require that the _SITE variable be assigned with the :=
assignment operator. This is weird, because that would make it the only
variable to require that, but only when using the github helper, which
is even less obvious and would cause a lot of trouble...
The obvious fixup would seem to be to use $(PKG) instead, because that
already contains the upper-case package name that vcan be used as a
prefix to variables.
However, that does not work either, because we have a check that forbids
a trsailing slash in _SITE, check that is done in pacakge/pkg-generic,
inside the xxx-inner macro, during the $(eval) call.
And at that time, PKG is not yet defined, because it is only defined for
an actual recipe.
So we can't seem to have a workable solution. So, just revert the patch.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Marcin Nowakowski <marcin.nowakowski@imgtec.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently it is always required to add package version as an argument to
the github helper. Since the version is always defined as PKG_VERSION,
drop this argument and generate it automatically inside the helper
routine.
The github helper function is extended to support both 2 and 3 argument
variants (ie. either use the provided package version argument or
automatically substitute with PKG_VERSION if not available), which can
make the transition of the package files easier as well allows using the
3-argument variant outside of package definitions.
Signed-off-by: Marcin Nowakowski <marcin.nowakowski@imgtec.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
For a while already, our git downloaded tarballs are reproducible, so
we can actually check the hash for them.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Some packages download files (especially patches) with a ? in the
URL. The ? marks the query part of the URL. However, the downloaded
file still contains the ? but from then on it doesn't designate a
query part anymore. Therefore, when fetching from PRIMARY or
BACKUP site over http, the server will report a 404 Not Found.
To fix, we need to replace the ? with %3F. Obviously, this should
be done only when fetching from PRIMARY or BACKUP. For fetching
from the real upstream, the ? really does designate the query part.
Fixes#9371.
Reported-by: Johan Derycke <johanderycke@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[Thomas: replace 'SECONDARY' by 'BACKUP'.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
For packages that use a version control repository rather than a pre-made
tarball, the directory prefix used inside the tarball is currently
FOO_BASE_NAME, which can be 'foo' or 'host-foo'.
This means that the hash of such tarball will be different for target and
host packages, even though the contents are exactly the same. Hence, if the
hash file is created based on 'foo', and later a fresh build is made where
'host-foo' happens to be built before 'foo' (with a different config, for
example), the hash will be detected as incorrect and a new download is
started.
This problem does not affect many packages/users, due to the number of
conditions to be met:
- the package should be available for target _and_ host
- the package needs to use a VCS download method, e.g. git, hg, svn, ...
This does not include standard github downloads, which download a pre-made
archive.
- there should be a hash file containing the hash of the downloaded archive.
Since normally there is no hash file for packages with sources coming from
a version control system, this restricts even further. Some examples of
packages in this category that do have a hash file (but not necessarily
match the earlier conditions): expedite, vexpress-firmware, squashfs, ...
- the archive needs to be stored in a 'primary site' after initial archiving
and thus be downloaded later using a non-version-controlled method, like
wget or scp. This is because the version control download methods do not
receive a '-H' parameter pointing to the hash file and thus no hashes are
checked at all even if the file is present.
While packages matching the third condition could be considered to be 'wrong'
and need to be fixed, it does actually makes sense to have a hash file for
packages from version control, in particular if they are stored in a
primary site as mentioned in the last condition.
Regardless of any different opinions on the previous paragraph, it is also
not conceptually correct that a tarball of a package source can contain a
Buildroot-specific directory prefix 'host-'. Therefore, use
FOO_RAW_BASE_NAME instead of FOO_BASE_NAME when calling the dl-wrapper.
Example test scenario that exhibits the problem:
$ rm -rf /tmp/dl dl/squashfs-9c1db6d13a51a2e009f0027ef336ce03624eac0d.tar.gz
$ make qemu_x86_64_defconfig
$ make host-squashfs-dirclean host-squashfs-source
$ mkdir /tmp/dl
$ mv dl/squashfs-9c1db6d13a51a2e009f0027ef336ce03624eac0d.tar.gz /tmp/dl/
$ sed -i -e 's,BR2_PRIMARY_SITE=.*,BR2_PRIMARY_SITE="file:///tmp/dl",' \
-e '/BR2_PRIMARY_SITE/aBR2_PRIMARY_SITE_ONLY=y' .config
$ make host-squashfs-dirclean host-squashfs-source
Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Introduce a new package variable $(PKG)_DL_OPTS. When this variable
is defined, its value is passed to the downloader as options to
the underlying command. Packages can now retrieve archives from server
expecting logins and passwords, use referer url, proxy or specific
options for cloning a repository.
Signed-off-by: Romain Perier <romain.perier@free-electrons.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Add a new package variable that packages can set to specify that they
need git submodules.
Only accept this option if the download method is git, as we can not get
submodules via an http download (via wget).
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Aleksandar Simeonov <aleksandar@barix.com>
Tested-by: Matt Weber <matt@thewebers.ws>
Reviewed-by: Matt Weber <matt@thewebers.ws>
Tested-By: Nicolas Cavallari <nicolas.cavallari@green-communications.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When a download via a VCS method fails, Buildroot attempts the download
from the backup mirror (if any is set). Such a download is done with the
wget helper.
Given a package that has a .hash file for relase tarballs, which also
allows the user to use a random revision from a VCS, the normal download
wrapper will not check for hashes, on the assumption that maybe they are
not reproducible [*] (or user configurable).
However, when the download fails (for any reason: network error,
upstream removed the revision, old machine with incomplete set of CA
certificates...), the backup download now proceeds over with http (or
https) with the wget wrapper. The wget wrapper *always* checks for
hashes, and when a .hash file is present but no hash is found for the
download, this is considered an error.
However, when we download from the backup mirror for a main download that
should have been done with git/hq/svn/.., we in this situation have no hash
for the download. Thus, we should not fail on a missing hash for that
download. Same situation if a primary site is used.
Add a test for the site-method in the DOWNLOAD_INNER macro; if it was either
one of the VCS method, pass a one-off BR_NO_CHECK_HASH_FOR variable set to
contain the tarball to download.
Fixes issues like those reported by the Travis build bots:
https://travis-ci.org/buildroot/buildroot-defconfig-testing/jobs/123624879
[Peter: Move logic to beginning of DOWNLOAD_INNER so it also applies for BR2_PRIMARY_SITE]
Reported-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Following commit 95a572282e (pkg-infra: move the git download helper to a
script, 2014-07-02), move the comment describing the shallow clone trickery as
well. Merge this comment with the existing helper comment that was added in
7e40a1103a (support/download: convert git to use the wrapper, 2014-08-03).
Rename $($(PKG)_DL_VERSION) to ${cset} to match the helper code context.
Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Baruch Siach <baruch@tkos.co.il>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently, assigning to BR_NO_CHECK_HASH_FOR but referencing variable
assigned to later, fails. Here's a failing test-case, which is the
reduced test-case of how we handle BR_NO_CHECK_HASH_FOR for now (warning
added for test-case):
export BR_NO_CHECK_HASH_FOR
BR_NO_CHECK_HASH_FOR += $(XENOMAI_SOURCE)
XENOMAI_VERSION = 2.6.4
XENOMAI_SOURCE = xenomai-$(XENOMAI_VERSION).tar.bz2
$(warning BR_NO_CHECK_HASH_FOR='$(BR_NO_CHECK_HASH_FOR)')
all:
Run it with simply make:
$ make
BR_NO_CHECK_HASH_FOR=' '
Now, change the first line to read:
export BR_NO_CHECK_HASH_FOR =
And we now get:
$ make
BR_NO_CHECK_HASH_FOR=' xenomai-2.6.4.tar.bz2'
This new behaviour will be needed later for Xenomai, which handles the
version string in an unusual way, so we can exclude its custom versions.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Some of the SOURCE_CHECK_* macros are using a non-standard two-spaces
indentation. This commit switches them to use a single tab based
indentation, like in the rest of Buildroot.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Tested-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
[with git show -w]
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
The DL_MODE variable is now no longer used with any other value than
"DOWNLOAD", so it no longer makes sense to have this variable at
all. Therefore, this commit gets rid of it.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Tested-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
As part of moving to a package infrastructure based source-check
implementation, we are going to move away from the global DL_MODE
variable to select the behavior of the DOWNLOAD_INNER macro.
As a preparation to this, this commit makes the DOWNLOAD_INNER macro
take a third argument, which is the action to be done: either DOWNLOAD
or SOURCE_CHECK. For now, the DOWNLOAD macro passes $(DL_MODE) as this
third argument, in order to keep the existing behavior.
In addition, a SOURCE_CHECK macro is added, which calls DOWNLOAD_INNER
with the appropriate action. This macro will be used in the upcoming
package infra based implementation of source-check.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[tested by doing a "make source" on a randpackageconfig]
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Now that the external-deps implementation relies on the per-package
<pkg>-all-external-deps and <pkg>-external-deps targets and no longer
on the 'source' target with a custom DL_MODE, we can get rid of the
support for the SHOW_EXTERNAL_DEPS DL_MODE value in the pkg-download
logic.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
In very constrained cases, it might be needed to not fail if a hash is
missing. This is notably the case for custom external toolchains to be
downloaded, because we do have a .hash file for external toolchains,
but we obviously can not have hashes for all existing custom toolchains
(he, "custom"!).
So, add a way to avoid failing in that case.
>From the Makefile, we export the list of files for which not to check
the hash. Then, from the check-hash script, if no check was done, and
the file we were trying to match in in this exclusion list, we just exit
without error.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Tested-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
changes v6 -> v7:
- /beautify/ the pattern in the case clause
Changed v5 -> v6: (Arnout)
- fix the pattern in the case clause
Changes v4 -> v5:
- micro-optimisation, use case-esac instead of a for-loop (Arnout)
- typoes (Arnout)
Changes v3 -> v4:
- drop the magic value, use a list of excluded files (Arnout)
Changes v1 -> v2:
- fix typoes in commit log
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Tested-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
When downloading from a repository, we have no way to ensure the
reproducibility of the generated archives, so we can't check the hashes.
Do not specifiy a hash file in those cases.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Do not hard-code QUIET in our download commands, since it is handled in
the backends.
Suggested by Fabio.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Fabio Porcedda <fabio.porcedda@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
If doing a silent build (make -s -> QUIET=-q), silence all downloads,
by passing the -q flag downward to backends as well as to check-hash.
Change a printf to use the trace functions.
Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Instead of repeating the check in our download rules, delegate the check
of the hashes to the download wrapper.
This needs three different changes:
- add a new argument to the download wrapper, that is the full path to
the hash file; if the hash file does not exist, that does not change
the current behaviour, as the existence of the hash file is checked
for in the check-hash script;
- add a third argument to the check-hash script, to be the basename of
the file to check; this is required because we no longer check the
final file with the final filename, but an intermediate file with a
temporary filename;
- do the actual call to the check-hash script from within the download
wrapper.
This further paves the way to doing pre-download checks of the hashes
for the locally cached files.
Note: this patch removes the check for hashes for already downloaded
files, since the wrapper script exits early. The behaviour to check
localy cached files will be restored and enhanced in the following
patch.
[Thomas: fix minor typo in comment.]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Instead of repeating the same test again and again in all our download
rules, just delegate the check for an already downloaded file to the
download wrapper.
This clears up the path for doing the hash checks on a cached file
before the download.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
Reviewed-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Instead of relying on argument ordering, use actual options in the
download wrapper.
Download backends (bzr, cp, hg...) are left as-is, because it does not
make sense to complexify them, since they are almost very trivial shell
scripts, and adding option parsing would be really overkill.
This commit also renames the script to dl-wrapper so it looks better in
the traces, and it is not confused with another wrapper.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Fixes (or atleast stops the build with a saner description):
http://autobuild.buildroot.net/results/1a9/1a9643f8633db038d4fe5ca4a32e4d52e70a3a1a/
We're using the same sources for host and target downloads, so it makes
sense to use the same <pkg>.hash file as well to ensure the host version
gets verified without us having to maintain a seperate host-<pkg>.hash file.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The Buildroot coding style defines one space around make assignments and
does not align the assignment symbols.
This patch does a bulk fix of offending packages. The package
infrastructures (or more in general assignments to calculated variable
names, like $(2)_FOO) are not touched.
Alignment of line continuation characters (\) is kept as-is.
The sed command used to do this replacement is:
find * -name "*.mk" | xargs sed -i \
-e 's#^\([A-Z0-9a-z_]\+\)\s*\([?:+]\?=\)\s*$#\1 \2#'
-e 's#^\([A-Z0-9a-z_]\+\)\s*\([?:+]\?=\)\s*\([^\\]\+\)$#\1 \2 \3#'
-e 's#^\([A-Z0-9a-z_]\+\)\s*\([?:+]\?=\)\s*\([^\\ \t]\+\s*\\\)\s*$#\1 \2 \3#'
-e 's#^\([A-Z0-9a-z_]\+\)\s*\([?:+]\?=\)\(\s*\\\)#\1 \2\3#'
Brief explanation of this command:
^\([A-Z0-9a-z_]\+\) a regular variable at the beginning of the line
\([?:+]\?=\) any assignment character =, :=, ?=, +=
\([^\\]\+\) any string not containing a line continuation
\([^\\ \t]\+\s*\\\) string, optional whitespace, followed by a
line continuation character
\(\s*\\\) optional whitespace, followed by a line
continuation character
Hence, the first subexpression handles empty assignments, the second
handles regular assignments, the third handles regular assignments with
line continuation, and the fourth empty assignments with line
continuation.
This expression was tested on following test text: (initial tab not
included)
FOO = spaces before
FOO = spaces before and after
FOO = tab before
FOO = tab and spaces before
FOO = tab after
FOO = tab and spaces after
FOO = spaces and tab after
FOO = \
FOO = bar \
FOO = bar space \
FOO = \
GENIMAGE_DEPENDENCIES = host-pkgconf libconfuse
FOO += spaces before
FOO ?= spaces before and after
FOO :=
FOO =
FOO =
FOO =
FOO =
$(MAKE1) CROSS_COMPILE=$(TARGET_CROSS) -C
AT91BOOTSTRAP3_DEFCONFIG = \
AXEL_DISABLE_I18N=--i18n=0
After this bulk change, following manual fixups were done:
- fix line continuation alignment in cegui06 and spice (the sed
expression leaves the number of whitespace between the value and line
continuation character intact, but the whitespace before that could have
changed, causing misalignment.
- qt5base was reverted, as this package uses extensive alignment which
actually makes the code more readable.
Finally, the end result was manually reviewed.
Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Cc: Yann E. Morin <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This drastically simplifies the wget helper, as it no longer has to deal
with atomically saving the downloaded archive.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
(Tested by running 'make busybox-source')
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
