Currently, the formatting we impose on the _LICENSE variable requires
that we also use the rarely used := assignment operator, which makes
the _LICENSE variable the only variable that users have to write with
this operator.
This really departs from the simplicity and consistency of using the
append-assignment, which we use for every other variable.
This is because the append-assignment operator surreptiously
introduces a space between the original value and the appended one. But
we can use this knowledge, to match any instance of a space followed by
a comma, and turn it into a single comma.
This allows users to now have a consistent use of the '=' and '+='
operators we use everywhere else in .mk files.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Add an OpenRC service that starts and stops sysv-init scripts. We order
that script 'after local' so that it is started after all other native
openrc services.
Signed-off-by: Michał Łyszczek <michal.lyszczek@bofc.pl>
[yann.morin.1998@free.fr:
- don't propagate the micro optimisation for running .sh scripts
- use spaces, not TABs
- stop services in reverse order
- reword commit log
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, foo-depends only builds build dependencies. This means that
download and extract dependencies are not built.
First, but a minor point, this is inconsistent with foo-show-depends,
which does display all dependencies:
$ make host-gzip-show-depends
host-skeleton host-tar
$ make host-gzip-depends
# Only host-skeleton is built and installed
Second, and more important, it makes it more difficult to preapre a
debug build, like so:
$ make foo-depends
$ tar cf output.tar output
$ make foo
# bummer, broken
# edit foo.mk to try and fix it
$ rm -rf output; tar xf output.tar
# rince and repeat
Change foo-depends so that it really builds all the dependencies for
foo, bringing it on-par with foo-show-depends.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
We have some internal packages which automatically bake a version string
from the git checkout, and we usually combine these with a
*_OVERRIDE_SRCDIR. I would like to let Buildroot *not* skip the .git
directory when picking up sources from the local checkout. It turns out
that the existing mechanism (*_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS) only
supports adding to the exclude list because `rsync` simply uses the
first match from the provided filtering rules.
Solve this by using the user-provided values first. If they match, then
`rsync` won't exclude stuff based on the generic VCS exclude patterns.
Signed-off-by: Jan Kundrát <jan.kundrat@cesnet.cz>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
* package/pkg-generic.mk
Add <pkg>_INSTALL_INIT_OPENRC so packages can define their own steps
to install openrc service scripts.
* docs/manual/adding-packages-generic.txt
update documentation about new hook.
Signed-off-by: Michał Łyszczek <michal.lyszczek@bofc.pl>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
show-dependency-tree was introduced in this release cycle, as a way to
quickly and easily provide the dependency tree to graph-depends.
show-dependency-tree is no longer used, now that graph-depends has been
switched over to using the more versatile show-info.
Beside, show-dependency-tree has never been part of a release.
Drop it.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Sometimes, it is need to quickly get the metadata of a subset of
packages, without resorting to a full-blown JSON query.
Introduce a new per-package (and per-filesystem) foo-show-info rule,
that otputs a per-entity valid JSON blob.
Note that calling it for multiple packages and.or filesystems at once
will not generate a valid JSON blob, as there would be no separator
between the JSON elements:
$ make {foo,bar}-show-info
{ "foo": { foo stuff } }
{ "bar": { bar stuff } }
However, jq is able to absorb this, with its slurping ability, which
generates an array (ellipsed and manualy reformated for readability):
$ make {foo,bar}-show-info |jq -s . -
[
{ "foo": { foo stuff } },
{ "bar": { bar stuff } }
]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Currently, the DOWNLOAD macro is context-dependent and expects
the PKG variable to be set to the current package.
This is not so nice.
Change the macro to expect the upper-case package name as a
parameter, rather than expect it from a variable.
Adapt the caller accordingly.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Why do things simply, when we can do it complicated?
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, host-xz and host-lzip are built as soon as the
corresponding tools are not provided by the system, independently of
whether they are really needed by the Buildroot configuration. This is
particularly annoying for host-lzip, which is only needed for very few
packages.
This commit modifies the generic package infrastructure to only add
host-lzip and host-xz as dependencies when really needed.
Signed-off-by: Norbert Lange <nolange79@gmail.com>
[Thomas:
- improve commit log
- as suggested by Yann E. Morin, make the lzip case similar to the xz
case]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The %-graph-rdepends should be marked PHONY, as the %-graph-depends is.
The %-show-dependency-tree should also be a PHONY rule, because it does
not actually create a file. Furthermore, to avoid the "Nothing to be
done for .." message, give that rule an actual recipe that just does
nothing.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, when we tweak the .la files, we do so unconditionally on all
.la files, even those we already fixed in a previous run.
This has the nasty side effect that each .la file will be reported as
being touched by all packages that are installed after the package that
actually installed said .la file.
Since we can't easily know what files were installed by a package (that
is an instrumentation hook, and comes after the mangling), we use a
trick (like is done in libtool?): we do mangle all files, each into a
temporary location; if the result is identical to the input, we remove
the temporary, while if the result differs from the output, we move
the temporary to replace the input.
Reported-by: Nicolas Cavallari <nicolas.cavallari@green-communications.fr>
Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Nicolas Cavallari <nicolas.cavallari@green-communications.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, when we need to build the full dependency graph, we call make
to show the list of packages (make show-targets), and then call it again
and again iteratively while it returns new packages.
Since calling make will parse the whole set of our Makefiles, this takes
quite a bit of time (~4s each here), and the total can get pretty long.
However, make being make, already builds the whole dependency tree
information, so we can just ask for it.
Add a new top-level rule 'show-dependency-tree' that displays the whole
set of dependencies for all packages. For each package, its name, type
and version is displayed, then all the direct, first-level dependencies
are dumped. We choose a format that is not unlike the dot-graph format,
because it is both easy to read as a human, and easy to parse as a
machine:
foo: target 1.2.3
foo -> bar host-meh
bar: target virtual
bar -> buz
buz: target 2.3.4
buz ->
host-meh: host virtual
host-meh -> host-bleark
host-bleark: host 3.4.5
host-bleark ->
rootfs-meh: host
rootfs-meh -> host-bleark
To be noted: rootfs are currently reported as if they were 'host'
packages, to stay aligned with how graph-depends currently treats them.
Ideally, graph-depends could be enhanced to recognise them separately,
but that is another story.
For just plain defconfig, which is about the smallest config we can have
with an internal toolchain, we already have a seven-fold improvement
(with the graph-depends rule modified to not run the pdf generation, to
be able to just compare the tree generation):
$ time make graph-depends
real 0m27.344s
$ time make show-dependency-tree
real 0m3.848s
>From defconfig, C++, wchar, locales, ssp, and allyespackageconfig,
tweaked for even more packages (qt5 not qt4, luajit to avoid multi
providers, etc...), the timings are (graph-depends still modified to
not generate the pdf):
$ time make graph-depends
real 1m56.459s
$ time make show-dependency-tree
real 0m5.748s
There. I don't think those numbers need any explanation whatsoever;
they do speak on their own. OK, for maths sake, the ratio is about
twenty-fold. So, "yeah", I guess... ;-)
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Calling to the graph-depends script is very costly, as it calls back to
'make' a lot of time.
It turns out that we already have the list of recursive dependencies, so
we can just print that.
As for the recursive reverse dependencies, we use the same memoisation
technique to cut-down on the expansion cost, which would otherwise be on
the order of 𝑶(𝑛²) (with 𝑛 enabled packages).
>From a defconfig, modified to use glibc, C++, wchar, locales, ssp, and
allyespackageconfig (tweaked to avoid multi providers, etc...), the
timings for X-show-recursive-rdepends are:
before after speedup #rdeps
libnss 0m22.932s 0m5.775s 3.97x 3
qt5base 0m41.176s 0m5.781s 7.12x 67
libjpeg 0m56.185s 0m5.749s 9.71x 228
libxml2 0m54.964s 0m5.795s 9.48x 271
freetype 0m46.754s 0m5.819s 8.07x 287
libpng 0m53.577s 0m5.760s 9.30x 303
sqlite 1m15.222s 0m5.807s 12.95x 801
libopenssl 1m25.471s 0m5.844s 14.63x 931
readline 1m13.805s 0m5.775s 12.78x 958
libzlib 1m11.807s 0m5.820s 12.34x 1039
toolchain 1m23.712s 0m6.080s 13.77x 2107
skeleton 1m27.839s 0m6.293s 13.96x 2111 (+1)
host-skeleton 1m27.405s 0m6.350s 13.76x 2172 (+2)
- speedup: ratio before/after
- #rdeps: number of recursive reverse dependencies, with the extra
dependencies returned with this patch, see below for the
reason.
So, for a low-level package with a lot of reverse dependencies, like
libzlibz, libopenssl or readline are, the timings are already very much
in favour of the change. This is less impressive with packages that
have few dependencies (libnss), but still much faster.
Also, remember that the config tested has as much packages enabled as
possible, so is in itself a degenerate case. With simpler and more
realistic configurations, the gains would probably be a bit lower than
reported above, but various tests still report good improvements
overall (note: coming up with a 'realistic' configuration is pretty
hard, as everyone and their dog have their notion of what is realistic
in their context, so nothing displayed here; timings are left as an
exercise for the interested parties to report aggravation in their
cases should they notice some regression).
Note that, more recursive reverse dependencies may be displayed now,
since we do not apply the exceptions applied in graph-depends. For
example, host-skeleton gains two new recursive reverse dependencies:
skeleton and toolchain, which are both exceptions in graph-depends.
As for direct (not reverse) dependencies: the gain is not as fantastic
as for reverse ones, but it is still noticeable, especially thanks to
a21212fb7c (package/pkg-generic: speed up RECURSIVE_FINAL_DEPENDENCIES);
just a few examples for %-show-recursive-depends:
before after speedup #deps
libzlib 0m46.864s 0m5.902s 7.94x 17
qt5base 0m57.590s 0m5.848s 9.85x 190
sqlite 0m46.601s 0m5.816s 8.01x 24
Basically, displaying recursive dependencies, direct or reverse, is
almost a constant now: it only slightly varies by about 10% depending
on the complexity of the dependency chain, with the parsing of the
Makefiles still accounting for the large majority of the time.
(PS. Thanks to Joseph for suggesting a list of interesting packages
to test, and thanks to Trent for his example of memoisation!)
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Joseph Kogut <joseph.kogut@gmail.com>
Cc: Trent Piepho <tpiepho@impinj.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Evaluating all the <PKG>_RECURSIVE_FINAL_DEPENDENCIES variables
(abbreviated RFD hereafter) ends up being quite slow. Enough, on a
reasonable modern workstation, to increase the time it takes to run
"make printvars" from 13 seconds in 2018.02 to 371 seconds in 2019.02.
This patch improves this by using dynamic programming to speed the
evaluation of RFD, reducing the before mentioned printvars time to about
14.6 seconds.
The evaluation of PKG1_RFD requires recursively evaluating each of
PKG1's dependencies' RFDs, then their dependencies' RFDs, and so on.
The same is done for PKG2_RFD. But it's likely that many of the
dependencies of PKG2 are the same as PKG1. And when we consider all
packages, the dependencies are re-computed many thousands of times.
To avoid this re-computation we memoize, or save, the computed value of
each RFD variable when it found the first time. Subsequent evaluations
re-use the memoized value.
Surprisingly, this ends up being not all the hard to implement in make.
The basic construct is this:
VAR = $(if !defined(VAR__X),$(eval VAR__X := value))$(VAR__X)
The first time VAR is evaluated VAR__X will not be defined, and code to
set VAR__X to the computed value is eval'd. Then the now defined value
of VAR__X is returned. Subsequent evaluations can just return VAR__X.
It is important to note that VAR is defined with '=', as not enough
information (namely, all packages' dependencies) is know when it is
parsed to find the correct value. VAR will be evaluated each time it is
used. But VAR__X is defined with ":=", so that it is evaluated once
when defined, and not each time it is used.
Signed-off-by: Trent Piepho <tpiepho@impinj.com>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
In commit 7fb6e78254 (core/instrumentation: shave minutes off the
build time), the built stampfile is used as a reference to detect files
installed by a package.
However, packages may install files keeping their mtime intact, and we
end up not detecting this. For example, the internal skeleton package
will install (e.g.) /etc/passwd with an mtime of when the file was
created in $(TOP_DIR), which could be the time the git repository was
checked out; that mtime is always older than the build stamp file, so
files installed by the skeleton package are never accounted for to that
package, or to any other package for that matters.
We switch to an alternate solution, which consists of storing some extra
metadata per file, so that we can more reasily detect modifications to
the files. Then we compare the state before the package is installed (by
reusing the existing list) and after the package is installed, compare
that to list any new file or modified files (in reality, ignoring
untouched and removed files). Finally, we store the file->package
association in the global list and store the new stat list as the global
list.
The format used for the .stat file is:
mtime:inode:perms:filetype:size,filename
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Trent Piepho <tpiepho@impinj.com>
[Peter: rename files, reformat, only look for files and symlinks and pass
LC_ALL=C to comm as pointed out by Thomas De Schampheleire]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
realpath is missing on oldish distributions, like Debian 7, which is
still used in the wild.
Use readlink instead; that has been available since the dawn of ages now
(well, coreutils had it in 2003).
Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In case a config script is called from a relative path, the $(dirname
$0) would return a relative path too.
Those paths are usually parts of includes or libraries search
directories, and the packagfes buildsystems may chdir() anywhere, and
thus the relative path will no longer be valid. For example:
$ ./host/powerpc-buildroot-linux-uclibc/sysroot/usr/bin/net-snmp-config --cflags
[...] -I./host/powerpc-buildroot-linux-uclibc/sysroot/usr/bin/../../.././bin/../powerpc-buildroot-linux-uclibc/sysroot/usr/include/libnl3 [...]
Canonicalise the path to be sure we use absolute paths.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This commit adjusts the logic in pkg-generic.mk that tweaks the
*-config shell scripts installed by various libraries to make it
compatible with per-package directories.
This requires two fixes:
- replacing $(STAGING_DIR) with a relative path from the config script
to the staging directory, rather than using an absolute path of the
staging directory.
Without this, a *-config script provided by package A, but called
from package B per-package directory will return paths from package A
per-package directory:
$ ./output/per-package/mcrypt/host/usr/<tuple>/sysroot/usr/bin/libmcrypt-config --libs
-L..../output/per-package/libmcrypt/host/usr/<tuple>/sysroot/usr/lib/
The libmcrypt-config script is installed by the libmcrypt package,
and mcrypt is a package that depends on libmcrypt. When we call the
libmcrypt-config script from the mcrypt per-package directory, it
returns a -L flag that points to the libmcrypt per-package
directory.
One might say: but this is OK, since the sysroot of the libmcrypt
per-package directory also contains the libmcrypt library. This is
true, but we encounter a more subtle issue: because -L paths are
considered before standard paths, ld ends up finding libc.so in the
libmcrypt per-package directory. This libc.so file is a linker
script that looks like this:
GROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a AS_NEEDED ( /lib/ld-linux.so.3 ) )
Normally, thanks to ld sysroot awareness, /lib/libc.so.6 in this
script is re-interpreted according to the sysroot. But in this
case, the library is *outside* the compiler sysroot. Remember: we
are using the compiler/linker from the "mcrypt" per-package
directory, but we found "libc.so.6" in the "libmcrypt" per-package
directory.
This causes the linker to really use the /lib/libc.so.6 from the
host machine, obvisouly leading to a build failure such as:
output/per-package/libgcrypt/host/opt/ext-toolchain/bin/../lib/gcc/nios2-linux-gnu/7.3.1/../../../../nios2-linux-gnu/bin/ld: cannot find /lib/libc.so.6
output/per-package/libgcrypt/host/opt/ext-toolchain/bin/../lib/gcc/nios2-linux-gnu/7.3.1/../../../../nios2-linux-gnu/bin/ld: cannot find /usr/lib/libc_nonshared.a
output/per-package/libgcrypt/host/opt/ext-toolchain/bin/../lib/gcc/nios2-linux-gnu/7.3.1/../../../../nios2-linux-gnu/bin/ld: cannot find /lib/ld-linux-nios2.so.1
- Some *-config scripts, such as the apr-1-config script, contain
references to host tools:
CC=".../output/per-package/apr/hosr/bin/arm-linux-gcc"
CCP=".../output/per-package/apr/hosr/bin/arm-linux-cpp"
We also want to replace those with proper relative paths. To
achieve this, we need to also replace $(HOST_DIR) with a relative
path. Since $(STAGING_DIR) is inside $(HOST_DIR), the first
replacement of $(STAGING_DIR) by @STAGING_DIR@ is no longer needed:
replacing $(HOST_DIR) by @HOST_DIR@ is sufficient. We still need to
replace @STAGING_DIR@ by the proper path though, as we introduce
@STAGING_DIR@ references in exec_prefix and prefix variables, as
well as -I and -L flags.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
In the current code, the creation of the main output directories
(BUILD_DIR, STAGING_DIR, HOST_DIR, TARGET_DIR, etc.) is done by a
global "dirs" target. While this works fine in the current situation,
it doesn't work well in a context where per-package host and target
directories are used.
For example, with the current code and per-package host directories,
the output/staging symbolic link ends up being created as a link to
the per-package package sysroot directory of the first package being
built, instead of the global sysroot.
This commit reworks the creation of those directories by having the
package/pkg-generic.mk code ensure that the build directory, target
directory, host directory, staging directory and binaries directory
exist before they are needed.
Two new targets, host-finalize and staging-finalize are added in the
main Makefile to create the compatibility symlinks for host and
staging directories. They will be extended later with additional logic
for per-package directories.
Thanks to those changes, the global "dirs" target is entirely removed.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Recently, some hash mismatch have been reported, both by users as well
as autobuilder failures, about tarballs generated from git repositories.
This turned out to be caused by users having the 'gzip' command somehow
aliased to 'pigz' (which stand for: parallel implementation of gzip,
which takes advantage of multi-processor system to parallelise the
compression).
Unfortunately, the output of pigz-compressed archives differ from that
of gzip (even though they *are* valid gzip-compressed streams).
Add a dependency check that ensures that gzip is not pigz. If that is
the case, define a conditional dependency to host-gzip, that is used as
a download dependency for packages that will generate compressed files,
i.e. cvs, git, and svn.
Fixes:
http://autobuild.buildroot.org/results/330/3308271fc641cadb59dbf1b5ee529a84f79e6d5c/
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Marcin Niestrój <m.niestroj@grinn-global.com>
Cc: Erico Nunes <nunes.erico@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The commands like "make show-build-order" or "make
<package>-show-build-order" show the build order and then print
"make[1]: Nothing to be done for 'show-build-order'" to stdout. It
pollutes output. Technically this message is true but it's not true
for user because he gets an information.
The <package>-show-build-order targets use $(info) for package name
printing. The make utility doesn't consider the internal directive as
a command so it think that it's "Nothing to be done". The patch adds
the empty command to <package>-show-build-order to inform make utility
that taget makes some real actions.
Signed-off-by: Serj Kalichev <serj.kalichev@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Thomas: invert $(info) and @:, as suggested by Yann.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Three of our download backends need a host tar that can generate
reproducible archives: cvs, git, and svn. The other two, bzr and hg,
use their internal implementation.
So, for those three that need it, and a dependency on host-tar when the
system tar is not appropriate.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
For some packages, we may need to have a certain set of host-tools built
before the download of said packages are attempted. For example, when
the system tar is not suitable, we will want to build our own tar before
we attempt a git download (because we generate a tarball in the git
backend).
Mimick the _EXTRACT_DEPENDENCIES, and introduce _DOWNLOAD_DEPENDENCIES.
As for _EXTRACT_DEPENDENCIES, we do not document _DOWNLOAD_DEPENDENCIES,
on the assumption that it is mostly for internal use.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In the pkg-inner macros, all variables, but the positional arguments,
must be $$-prefixed, so that they are expanded only when the macro is
evaluated in each package, not when the macro is parsed.
It is to be noted, though, that the current code, even though
incorrect by the above rules, seemed to work. However, the upcoming
addition of download dependencies, mimicking that code, would not work
unless it was $$-prefixed.
So, for consistency sake, and for correctness sake, let's always use
the $$-prefix in the inner macro.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This adds one column to the legal-info manifest table. It contains the
dependencies of the given package and their licenses. This information
is useful when assessing license compatibility of the packages and
their libraries.
An example of the content of the new column for the MPD package is
shown below:
"alsa-lib [LGPL-2.1+ (library), GPL-2.0+ (aserver)] boost
[BSL-1.0] libid3tag [GPL-2.0+] libmad [GPL-2.0+] libogg
[BSD-3-Clause] libvorbis [BSD-3-Clause] libzlib [Zlib]
skeleton-init-common [unknown] skeleton-init-sysv [unknown] sqlite
[Public domain] toolchain-external-linaro-arm [unknown]"
[Credits to Yann E. MORIN <yann.morin.1998@free.fr> for suggesting a
few simplifications.]
Signed-off-by: Michal Sojka <sojka@merica.cz>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The last parameter {HOST|TARGET} is now first. With this change,
adding new columns to the legal manifest file (as in the next commit)
will be slightly easier to review.
Signed-off-by: Michal Sojka <sojka@merica.cz>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When a package has a version selection (e.g. Qt5), the licensing terms
may be different across versions, but lie in similarly named files (e.g.
'LICENSE').
However, when we check a file, all the hashes for it must match. So, we
can't have the hashes for two different content of the same file. We
overcame that limitation in the legal-license-file macro, which checks
whether a package has a .hash file in a versioned subdir.
For consistency, we would like to also store the source hashes in that
per-version subdir.
Rather than reconstruct the path to the hash file everywhere we need it,
add a variable that points to it.
Existing users will be converted over in followup patches.
Note: the check for a missing hash file is done in the check-hash helper
script, so this variable must always yield a filename, even of a missing
file, thus we do not use $(wildcard...) to resolve the hash file path;
we use $(wildcard...) only to check if the versioned .hash file exists.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Baruch Siach <baruch@tkos.co.il>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, the timestamps that we keep in build-time.log use a
second-level precision. However, as we are going to introduce a new
type of graph to draw the time line of a build, this precision is
going to be insufficient, as a number of steps are so short that they
are not even one second long, and generally the rounding to the second
gives a not so great looking graph.
Therefore, we add to the timestamps the nanoseconds using the %N date
specifier. A milli-second precision would have been sufficient, but %N
is all what date(1) provides at the sub-second level.
Since this is changing the format of the build-time.log file, this
commit adjusts the support/scripts/graph-build-time script
accordingly, to account for the floating point numbers that we have as
timestamps.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, when the site method is explicitly set to a SCM other than
git, the main download is correctly excluded from being hash-checked.
But when the site method is inferred from the site uri, the download
from a SCM other than git is wrongly being hash-checked.
Fix this by moving the code that excludes SCM methods from hash-check
below the code that infers the site method.
Currently there is no package in the tree that uses inferred site
method, and that is why the autobuilder didn't caught this. We had
packages using inferred site method in the past, the last one was
'expect', but since they didn't have a hash file (for a license or
extra-download for example) the build didn't error out.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The 'local' site method is easily confused with the 'file' site method,
making people create packages like this:
FOO_SITE_METHOD = local
FOO_SOURCE = foo.tar.gz
$(eval $(generic-package))
Due to the intricacies of the generic package infra, this does not
cause an error; instead, the foo.tar.gz tarball that happens to be
present in the download directory will be used. This behaviour differs
greatly from what is specified in the manual.
Instead, error out immediately if a package specifies the 'local' site
method but does not specify a _SITE.
We check for _OVERRIDE_SRCDIR rather than checking for _SITE, just
after _OVERRIDE_SRCDIR has been set to _SITE. Indeed, a package that
sets _OVERRIDE_SRCDIR but not _SITE currently works correctly. There is
no reason to make it fail.
See also
https://stackoverflow.com/questions/50364655/including-patches-to-build-root
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Also call the step hooks from the three steps they are currently not
called in:
- download,
- actual download (when main archive is not the real source, like
external toolchains),
- rsync (for local or override-srcdir).
Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Commit ea55e1323 (core/pkg-infra: don't enforce site-method for extra
downloads) forgot to account for those packages that have nothing to
download, like the skeleton, or like virtual packages...
The side effect is that the message "foo Downloading" is thus
displayed when it should not be.
Fix that.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The site method is stored in FOO_SITE_METHOD, not in FOO_SITE.
Fixes:
http://autobuild.buildroot.net/results/131/13196dd779bc9e3b172c74851546dd4c4752aa02/
[Peter: add autobuilder reference]
Reported-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The site method only ever applies to the main download, while extra
downloads are always to be fetched with wget.
However, the site method is prepended to the URL from within the
DOWNLOAD macro (well, a variable evaluated in the DOWNLOAD macro),
which is called for each download of a package, thus effectively
prepending the site method to all downloads, even the extra ones (and
the patches).
We fix that by prepending the site method from within the
generic-package infra, so that it only applies to the main download.
For that, we move the main _SOURCE out of the foreach loop, so that
we can prepend the site-method to it, without impacting the other
downloads.
Reported-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, when the main download if from an SCM, we exclude all
downloads from being hash-checked, on the assumption that we don't
have hashes for downloads from an SCM.
However, the exclusion is computed on the DOWNLOAD macro, which is
called for each download of a package, thus effectively disabling
hash checks for extra downloads, even though those are only ever
download with wget.
What we really wanted to do, in fact, was to exclude just the main
download.
We fix that by appending the main source file to the global list of
excluded files, from within the generic-package infra itself.
Reported-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When rebuilding a package, simply appending the package's file list to
the global list means that the package list grows for every rebuild, as
does the time taken to check for files installed by multiple packages.
Furthermore, we get false positives where a file is reported as being
installed by multiple copies of the same package.
With this approach we may end up with orphaned files in the target
filesystem if a package that has been updated and rebuilt no longer
installs the same set of files, but we know that only a clean build will
produce reliable results. In fact it may be helpful to identify these
orphaned files as evidence that the build is not clean.
Signed-off-by: John Keeping <john@metanate.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Update the documentation accordingly.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
[yann.morin.1998@free.fr: slight rephrasing in error message, update manual]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Fixes:
http://autobuild.buildroot.net/results/b4a/b4af0de4ae9630ccbe7890f69047f216f2ff5119/
With the change to the DOWNLOAD macro, packages using FOO_ACTUAL_SOURCE_TARBALL fails:
>>> skeleton-init-common Collecting legal info
sourceryg++-2017.05-4-nios2-linux-gnu.src.tar.bz2: OK (md5: 529a7fecf33d0d113a446413b9d1e173)
sourceryg++-2017.05-4-nios2-linux-gnu.src.tar.bz2: OK (sha256: 6e65878d0453708ee19098d3d68985bda244938d35999f3859915a2f5574fa08)
/bin/bash: line 1: @mkdir: command not found
package/pkg-generic.mk:148: recipe for target '/accts/mlweber1/rclinux/rc-buildroot-test/scripts/instance-1/output/build/toolchain-external-codesourcery-niosII-2017.05-4/.stamp_actual_downloaded' failed
Which is caused by the continuation character '\'. This has been present
since the make target was introduced in commit eace9d6133
(core/legal-info: ensure legal-info works in off-line mode). It isn't clear
to me why it was done like that, but it fails with the DOWNLOAD macro
rework, so drop it.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This per package variable can be used to specify the download
subdirectory used by that package.
The use case here is for example linux-headers and linux, which share
the same sources (because they are the same upstream project), so we
don't want to download twice the kernel, nor store it multiple times
either.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
With all the previous changes, we are now ready to add a subdirectory to
the DL_DIR.
The structure will now be DL_DIR/PKG_NAME/{FILE1,FILE2}
This is needed for multiple reasons:
- Avoid patches with name like SHA1.patch laying flat in DL_DIR,
which makes it hard to know to which packages they apply
- Avoid possible collisions if two releases have the same name
(e.g: v01.tar)
- Allow the possibility to handle a git cache per package in the
newly created subdirectory.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Let the infrastructure use the already existing variable $(PKG)_DL_DIR
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This commit adds the support for <pkg>-show-recursive-depends and
<pkg>-show-recursive-rdepends which respectively show the list of all
dependencies or reverse dependencies for a given package. The existing
show-depends and show-rdepends only show the first-level dependencies,
while show-recursive-depends and show-recursive-rdepends show
recursively the dependencies.
It is worth mentioning that while show-recursive-depends really shows
all dependencies, show-recursive-rdepends is a bit limited because the
reverse dependencies of host packages are not properly accounted
for. But that's a limitation that already exists in show-rdepends, and
that cannot easily be solved.
Signed-off-by: George Redivo <george.redivo@datacom.ind.br>
[Thomas:
- split from the patch that was also changing graph-depends
- rename show-rrdepends to show-recursive-rdepends
- add show-recursive-depends
- don't create GRAPHS_DIR.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This allows using <PKG>_SRCDIR_OVERRIDE_RSYNC_EXCLUSIONS in local.mk to
skip copying parts of source trees unneeded for building. For example,
when developing WebKitGTK+, it's handy to skip copying all the tests and
other build directories, which are huge:
WEBKITGTK_OVERRIDE_SRCDIR = /home/aperez/WebKit
WEBKITGTK_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS = \
--exclude JSTests --exclude ManualTests \
--exclude PerformanceTests --exclude WebDriverTests \
--exclude WebKitBuild --exclude WebKitLibraries \
--exclude WebKit.xcworkspace --exclude Websites \
--exclude Examples
This saves a good chunk of time when rsync is used for the first time to
copy the source tree over before building.
Signed-off-by: Adrian Perez de Castro <aperez@igalia.com>
[Arnout: move documentation to the end of the section]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
In current Buildroot, clashes occur between the variables _NAME and
_BASE_NAME for two packages called foo and foo-base, i.e.
Package foo:
FOO_NAME = foo
FOO_BASE_NAME = foo-1.2.3
Package foo-base:
FOO_BASE_NAME = foo-base
FOO_BASE_BASE_NAME = foo-base-4.5.6
where variable FOO_BASE_NAME is clashing between these two packages.
Specific cases where this clash is already existing are:
- alljoyn-base
- alljoyn-tcl-base
- perl-xml-sax-base
The problem is generic and can occur for a number of variables in Buildroot.
A non-exhaustive list:
<pkg>_BASE and <pkg>_BASE_NAME
<pkg>_BASE_NAME and <pkg>_RAW_BASE_NAME
<pkg>_DIR and <pkg>_DL_DIR
<pkg>_VERSION and <pkg>_DL_VERSION
<pkg>_SOURCE and <pkg>_TARGET_SOURCE
<pkg>_INSTALL_IMAGES and <pkg>_TARGET_INSTALL_IMAGES (same for _STAGING and _TARGET)
<pkg>_LICENSE_FILES and <pkg>_MANIFEST_LICENSE_FILES
<pkg>_DEPENDENCIES and <pkg>_FINAL_DEPENDENCIES
One solution is to use another separator than '_' to separate the
package name from the rest of the variable name. For example, a double
underscore:
FOO__NAME
FOO__BASE_NAME
FOO_BASE__NAME
FOO_BASE__BASE_NAME
However, making that change for only this case means that the variable
naming is no longer consistent. And making the change for all variables has
a large impact, also on certain user scripts.
For now, keep it simple, and rename FOO_BASE_NAME into FOO_BASENAME, so that
the variables become:
FOO_NAME
FOO_BASENAME
FOO_BASE_NAME
FOO_BASE_BASENAME
For consistency, also adapt FOO_RAW_BASE_NAME. Since FOO_RAW_BASENAME would
still pose a conflict with a package called 'foo-raw', take the opportunity
to rename it into FOO_BASENAME_RAW instead, which does not pose a conflict
as we have no variable called FOO_RAW.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Sam Voss <sam.voss@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Now that the pre-fs ones are run on a transient copy of target/, the
post-fs hooks are no longer needed because we no longer need to restore
the target/ directory as it is only a internal copy.
Remove support for the post-fs hooks, and update the sole package using
them.
We do not add a legacy check because this was mostly a purely-internal
detail that was never really exposed nor documented.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Like we do for packages with the PKG variable, set ROOTFS to contain the
upper-case name of the rootfs currently being generated.
This will be useful in later patches, when we need more per-rootfs
variables, like a per-rootfs TARGET_DIR for example.
In Makefiles, per-rule variables trickle down the dependency chain, to
all dependencies of that rule, so we have to stop ROOTFS as soon as
we're not in a rootfs. This means we have to stop it at target-finalize
(which is a dependency of all filesystems), and for each package
individually, since some packages (host or target) can be direct
dependencies of filesystems as well.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
