Commit graph

262 commits

Author SHA1 Message Date
Heiko Thiery 9ff7b61705 package/pkg-generic.mk: enable hash checks for svn tarbals
With commit 89f5e98932 support for
reproducible archives was added. Thus archives generated from svn do no
longer needs to be added to BR_NO_CHECK_HASH_FOR.

Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-05-25 23:20:15 +02:00
Yann E. MORIN 0aed4c2dae linux: allow packages to set kernel config options
Currently, the linux kernel will apply some fixups on its .config file,
based on whether some packages are enabled or not. That list of
conditional fixups is getting bigger and bigger with each new package
that needs such fixups, culminating with the pending firewalld one [0].

Furthermore, these fixups are not accessible to packages in br2-external
trees.

Add a new per-package variable, that packages may set to the commands to
run to fixup the kernel .config file, which is added at the end of the
linux' own fixups.

This opens the possibility to write things like;

    define FOO_LINUX_CONFIG_FIXUPS
        $(call KCONFIG_ENABLE_OPT,BLA)
    endef

Of course, it also opens the way to run arbitrary commands in there, but
any alternative that would be declarative only, such as a list of
options to enable or disable (as an example):

    FOO_LINUX_CONFIG_FIXUPS = +BAR -FOO +BUZ="value"

.. is not very nice either, and such lists fall flat when a value would
have a space.

For packages that we have in-tree, we can ensure they won't play foul
with their _LINUX_CONFIG_FIXUPS. For packages in br2-external trees,
there's nothing we can do; users already have the opportunity to hack
into the linux configure process by providing LINUX_PRE_CONFIGURE_HOOKS
or LINUX_POST_CONFIGURE_HOOKS anyway...

.. which brings the question of why we don't use that to implement the
per-package fixups. We don't, because _PRE or _POST_CONFIGURE_HOOKS are
run after we run 'make oldconfig' to sanitise the mangled .config.

[0] http://lists.busybox.net/pipermail/buildroot/2020-March/278683.html

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Adam Duskett <aduskett@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-05-01 15:50:28 +02:00
Yann E. MORIN ee772f3256 package/pkg-generic.mk: don't set INSTALL_{TARGET, STAGING, IMAGES} for host packages
By their very nature, host packages have no target, staging, or images
install steps; they have a single install step, that is always
performed.

As such, setting the corresponding _INSTALL_{TARGET,STAGING,IMAGES}
variables does not make sense for host packages.

However, people (and scripts) may get confused when they process the
output of printvars, e.g.:

    $ make printvars VARS=HOST_LIBTOOL_INSTALL_TARGET
    HOST_LIBTOOL_INSTALL_TARGET=YES

Only set those variables for target packages. There is no
corresponding variable for host packages, as they are always installed
(and only once).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: eeppeliteloop@gmail.com
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-25 14:56:59 +02:00
Thomas Petazzoni 7ae7c82dd6 package/pkg-generic.mk: also replace /lib by STAGING_DIR/lib in .la files
After the staging installation, we replace a number of paths in libtool
.la files so that those paths point to STAGING_DIR instead of a location
in the build machine.

However, we replace only paths that start with /usr. And it turns out
that the linux-pam package is configured with --libdir=/lib (linux-pam
seems to always be installed in /lib rather than /usr/lib).

Due to this, libpam.la contains the following line:

  libdir='/lib'

When building a configuration that has:

 - BR2_ROOTFS_MERGED_USR=y
 - BR2_PACKAGE_LINUX_PAM=y
 - BR2_PACKAGE_POLKIT=y

on a system that has its system-wide PAM library installed in /lib,
the build fails with:

/lib/libpam.so: file not recognized: File format not recognized

For some reason, libtool searches only in STAGING_DIR/usr/lib, but
when BR2_ROOTFS_MERGED_USR=y, STAGING_DIR/lib points to
STAGING_DIR/usr/lib, so libtool finds libpam.la. And this libpam.la
contains a bogus libdir='/lib' path. libtool then goes on, finds
/lib/libpam.so, and links with it, causing the build failure.

By doing the proper replacement of libdir='/lib', we have a correct
libpam.la, and solve the build issue.

There is no autobuilder failure associated to this issue, as it
requires /lib/libpam.so to exist. This is the case on ArchLinux, on
which Xogium reported the issue, which can also be reproduced in an
ArchLinux container.

Reported-by: Xogium <contact@xogium.me>
Cc: Xogium <contact@xogium.me>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Yann E. MORIN <yann.morin.1998@free.fr>
[yann.morin.1998@free.fr:
  - tested by manually creating a symlink to libpam.so in /lib
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-04-05 23:17:26 +02:00
Thomas De Schampheleire 9f876c7f16 Makefile: fix package file list if FOO_SUBDIR is set
If a package sets a FOO_SUBDIR (meaning its sources are not under
output/build/foo-123 but under output/build/foo-123/$(FOO_SUBDIR)), the
.files-list.txt file were also created under FOO_SUBDIR, due to which the
logic in the Makefile would not find it.

Change the instrumentation steps so that the file list is directly under the
package dir, ignoring the subdir.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-03-20 22:15:08 +01:00
Thomas Petazzoni c84ce1f98c package/pkg-generic.mk: create directories before calling hooks
In commit 0e2be4db8a
("package/pkg-generic: make file list logic parallel build
compatible"), the logic to create the list of files installed by a
particular package was significantly reworked to be compatible with
top-level parallel build.

Before this commit, there was only a after-install step of listing the
files in HOST_DIR/TARGET_DIR/STAGING_DIR. But after this commit, we
now have a before-install logic and an after-install logic.

It turns out that when the before-install logic is called for the very
first host package, $(HOST_DIR) doesn't exist yet, and therefore the
cd $(2) fails, with an error message:

/bin/sh: line 0: cd: /home/thomas/buildroot/output/host: No such file or directory

In fact, $(HOST_DIR), $(STAGING_DIR), $(TARGET_DIR) and
$(BINARIES_DIR) are created by the make rules for host installation,
staging installation, target installation and images installation, but
*after* calling the step_start hooks.

So, we simply fix this problem by creating the directories *before*
calling the step_start hooks.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-03-17 21:10:09 +01:00
Thomas Petazzoni 15e96f9417 package/pkg-generic.mk: in image install, print message before pre-hooks
In all steps, we print the message indicating the start of the step
using the MESSAGE macro before running pre-hooks. Except in the image
installation step, where the message is printed after the pre-hooks.

Let's fix this inconsistency.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-29 17:47:47 +01:00
Thomas Petazzoni 0e2be4db8a package/pkg-generic: make file list logic parallel build compatible
The current solution used to collect the list of files installed by
packages does not work for top-level parallel build. Indeed, we rely
on a file created after the installation of the previous package to
build the list of files installed by the current package.

This works well when packages are built sequentially, but badly fails
when using top-level parallel build.

More specifically, top-level parallel build can fail with:

comm: /home/thomas/buildroot/output/build/.files-list-host.new: No such file or directory

Because that file has been removed concurrently by the build process
of another package.

This commit reworks the logic in a very straight-forward way. Before
the installation of each package, we store the list of files that are
already installed and store it in the package build directory. After
the installation of each package, we store again that list of files,
calculate the difference with the before file, and store that as the
list of files installed by that package, still in the package build
directory.

At the end of the build, in target-finalize we collect all the
collected information into the global package file lists, that
continue to be installed in the same location as before, with the same
name.

There are however some differences:

 (1) The files are no longer ordered in build order, but by alphabetic
     ordering of packages. Indeed, "build order" no longer makes any
     sense in the context of top-level parallel build.

 (2) Some files which were incorrectly tracked are no longer
     tracked. For example, the toolchain package is a target package,
     but it installs files in $(HOST_DIR). In the previous logic, the
     files installed by the toolchain package in $(HOST_DIR) were
     incorrectly affected to the next host package that was installed
     after the toolchain package. With our new logic, those files are
     no longer tracked at all. To fix this, we would have to change
     the logic to scan HOST_DIR/TARGET_DIR/STAGING_DIR for all
     installation steps, not just for the install-host, install-target
     and install-staging steps respecitively. But the result was
     already incorrect anyway, and therefore this should be fixed
     separately.

Note that the check_bin_arch hook needs to be adjusted: it was using
the global package-file-list.txt file, but this file is now created
only at the very end of the build. So instead, we use the current
package .file-list.txt file to know which packages have been installed
by the current package in $(TARGET_DIR).

Fixes:

  http://autobuild.buildroot.net/results/4e60fa31b1cd08bc7fdf9c5dd3a3f4941e029ba3/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-27 22:46:20 +01:00
Thomas Petazzoni 894d58a15c package/pkg-generic.mk: simplify step_pkg_size
Use the same trick in step_pkg_size as the one used in check_bin_arch:
factorize the two $(filter ...) calls into one, checking in one step
the step and whether it's the beginning or end of the step.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-27 22:45:12 +01:00
Thomas De Schampheleire 509db3b88a core: fix packages-file-list.txt after an incremental build
The package instrumentation step 'step_pkg_size' is populating the files:
    output/build/packages-file-list.txt
    output/build/packages-file-list-staging.txt
    output/build/packages-file-list-host.txt
by comparing the list of files before and after installation of a package,
with some clever tricks to detect changes to existing files etc.

As an optimization, instead of gathering this list before and after each
package, where the 'after-state' of one package is the same as the
'before-state' of the next package, only the 'after-state' is used and
is shared between packages.

This works fine, except at the end of the build, as explained next.

In the target-finalize step, many files will be touched. For example, files
like /etc/hosts, /etc/os-release, but also all object files that are
stripped, and all files touched by post-build scripts or created by rootfs
overlays. This means that the 'after-state' of the last package does not
reflect the actual situation after target-finalize is run.

For a single complete build this poses no problem. But, if one incrementally
rebuilds a package after the initial build, e.g. with 'make foo-rebuild',
then all changes that happened in target-finalize at the end of the initial
build (the 'after-state' of the last package built) will be detected as
changes caused by the rebuild of package foo. As a result, all these files
will incorrectly be treated as 'owned' by package foo.

Correct this situation by capturing a new state at the end of
target-finalize, so that the 'before-state' of an incremental build will be
correct.

Note: the reasoning above talks about packages-file-list.txt and
target-finalize, but also applies to
packages-file-list-staging.txt/staging-finalize and
packages-file-list-host.txt/host-finalize.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-17 22:19:33 +01:00
Thomas Petazzoni 8bf95d3b55 package/pkg-generic.mk: also apply extractor-pkg-dependency to <pkg>_EXTRA_DOWNLOADS
For now, the extractor dependencies were only calculated for
<pkg>_SOURCE, so if the package manually downloads another file using
<pkg>_EXTRA_DOWNLOADS and then extracts it with $(call
suitable-extractor), we are missing the corresponding dependency on
the appropriate extracting tool.

Since the vast majority of <pkg>_EXTRA_DOWNLOADS are compressed files
that will be uncompressed at build time, it makes sense to derive the
corresponding extractor dependencies directly in the common package
infrastructure, rather than having each and every package using
<pkg>_EXTRA_DOWNLOADS making this effort.

On a system without xzcat, before this patch:

$ make printvars VARS=HOST_GETTEXT_TINY_EXTRACT_DEPENDENCIES
HOST_GETTEXT_TINY_EXTRACT_DEPENDENCIES=host-tar

After this patch:

$ make printvars VARS=HOST_GETTEXT_TINY_EXTRACT_DEPENDENCIES
HOST_GETTEXT_TINY_EXTRACT_DEPENDENCIES=host-tar    host-xz

This commit most notably fixes the build of host-gettext-tiny on
systems without xzcat, and with per-package support enabled. Indeed,
the main _SOURCE for gettext-tiny is a .gz file, but it has a .xz file
in its _EXTRA_DOWNLOADS, which is then extracted. Except that xzcat
being missing from the dependencies, it is not built.

Fixes:

  http://autobuild.buildroot.net/results/83c6d47c06334bef27791a59bdd491b1de124c49/

Suggested-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2019-12-17 22:09:53 +01:00
Thomas Petazzoni f24332cdbc package/pkg-generic.mk: use extractor-pkg-dependency macro
Instead of manually calculating the EXTRACT_DEPENDENCIES value based
on the archive extension, let's use the newly introduced
extractor-pkg-dependency macro.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2019-12-17 22:09:51 +01:00
Thomas Petazzoni 0b6a083d12 package/pkg-utils.mk: rename extractor-dependency to extractor-system-dependency
The extractor-dependency macro returns which system-provided tools are
needed to be able to extract the archive passed as argument. The
result of this macro is added to DL_TOOLS_DEPENDENCIES so that the
logic in support/dependencies/ verifies that the necessary tools are
provided by the system.

However, we are going to add another macro, extractor-pkg-dependency,
which says which Buildroot packages are needed to extract the archive
passed as argument. Indeed, for those archive types, if the extractor
is not provided system-wide, we build it as a host Buildroot package.

To clarify the distinction between the upcoming
extractor-pkg-dependency and existing extractor-dependency, we rename
the latter to extractor-system-dependency.

We take this opportunity to extend the documentation of this macro.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2019-12-16 22:59:16 +01:00
Thomas Petazzoni 97dee44a6c package/pkg-generic.mk, support/scripts/fix-rpath: fix per-package regexp
Commit c4e6d5c8be ("core: implement
per-package SDK and target") had a mistake on the regexp that is used
to match $(PER_PACKAGE_DIR)/<something>/, and due to this, the regexp
was never matched.

The + sign in [^/]+ which was suggested by Yann E. Morin during the
review of the per-package patch series (instead of [^/]*) needs to be
escaped to be taken into account correctly. Without this, the regexp
doesn't match, and the replacement is not done, causing:

 (1) For the libtool fixup in pkg-generic.mk, the lack of replacement
     causes libtool .la files to not be tweaked as expected, which it
     turn causes build failures reported by the autobuilder.

 (2) For the fix-rpath, the RPATH of host binaries in the SDK were not
     correct.

Interestingly, we have the same regexp in
support/scripts/check-host-rpath, but here the + sign does not need to
be escaped.

Fixes:

  http://autobuild.buildroot.net/results/d4d996f3923699e266afd40cc7180de0f7257d99/ (libsvg-cairo)
  http://autobuild.buildroot.net/results/56330f86872f67a2ce328e09b4c7b12aa835a432/ (bind)
  http://autobuild.buildroot.net/results/9e0fc42d2c9f856b92954b08019b83ce668ef289/ (ibrcommon)
  and probably a number of other similar issues

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-12-12 08:27:54 +01:00
Peter Korsgaard 7b24bd59c8 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-12-02 09:39:41 +01:00
Peter Korsgaard 0f5ac40374 package/{pkg-generic, python, python3}: rename .py file exclusion variable to not conflict
Fixes:
http://autobuild.buildroot.net/results/3b6/3b6280b0b7a9634b747db2865b21c6266007c725/

The PYTHON_KEEP_PY_FILES global variable conflicts with the per-package
<pkg>_KEEP_PY_FILES variable for the python package, causing make to
complain:

package/zlib/zlib.mk:7: *** Recursive variable 'PYTHON_KEEP_PY_FILES' references itself (eventually).  Stop.

As a workaround, rename the global variable to KEEP_PYTHON_PY_FILES so it
cannot conflict with the per-package variable.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-12-01 21:55:37 +01:00
Thomas Petazzoni 56f3ed3fc1 package/{pkg-generic, python, python3}: add mechanism to exclude .py files from removal
When BR2_PACKAGE_PYTHON{,3}_PYC_ONLY=y, we force remove all .py files
from the system, as they have all been byte-compiled into their .pyc
variants.

However, it turns out that some packages (e.g: OpenCV) do some funky
things with a few .py files: they pass them through Python's
execfile() facility, which only works with .py files and not .pyc
files. It is used by OpenCV for example to read two small
configuration files.

In order to support such use cases, this commit introduces a very
simple mechanism by which packages can exclude some path patterns from
the .py removal: a per-package <pkg>_KEEP_PY_FILES variable that is
collected into a global PYTHON_KEEP_PY_FILES variable, then used by
the python/python3 target-finalize hooks.

This variable is intentionally not documented, this is really a hack
that we ideally would like to see go away, and we'd rather not see its
usage spread too much.

This is necessary to be able to fix bug #12171.

[Peter: check if PYTHON_KEEP_PY_FILES contains non-white space]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-11-30 09:46:27 +01:00
Thomas Petazzoni b27fb7871c package/pkg-generic: make libtool .la files compatible with per-package directories
Libtool .la files unfortunately contain a number of absolute paths,
which now refer to per-package directories. Due to this, when building
package A, .la files may contain absolute paths referring to
directories in package B per-package sysroot. This causes some -L
flags referring to other sysroot from being added, which doesn't work
as the linker no longer realizes that such paths are within its
sysroot.

To fix this, we introduce a replacement step of .la files in the
configure step, to make sure all paths refer to this package
per-package directory.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-11-29 15:24:05 +01:00
Thomas Petazzoni c4e6d5c8be core: implement per-package SDK and target
This commit implements the core of the move to per-package SDK and
target directories. The main idea is that instead of having a global
output/host and output/target in which all packages install files, we
switch to per-package host and target directories, that only contain
their explicit dependencies.

There are two main benefits:

 - Packages will now see only the dependencies they explicitly list in
   their <pkg>_DEPENDENCIES variable, and the recursive dependencies
   thereof.

 - We can support top-level parallel build properly, because a package
   only "sees" its own host directory and target directory, isolated
   from the build of other packages that can happen in parallel.

It works as follows:

 - A new output/per-package/ directory is created, which will contain
   one sub-directory per package, and inside it, a "host" directory
   and a "target" directory:

   output/per-package/busybox/target
   output/per-package/busybox/host
   output/per-package/host-fakeroot/target
   output/per-package/host-fakeroot/host

   This output/per-package/ directory is PER_PACKAGE_DIR.

 - The global TARGET_DIR and HOST_DIR variable now automatically point
   to the per-package directory when PKG is defined. So whenever a
   package references $(HOST_DIR) or $(TARGET_DIR) in its build
   process, it effectively references the per-package host/target
   directories. Note that STAGING_DIR is a sub-dir of HOST_DIR, so it
   is handled as well.

 - Of course, packages have dependencies, so those dependencies must
   be installed in the per-package host and target directories. To do
   so, we simply rsync (using hard links to save space and time) the
   host and target directories of the direct dependencies of the
   package to the current package host and target directories.

   We only need to take care of direct dependencies (and not
   recursively all dependencies), because we accumulate into those
   per-package host and target directories the files installed by the
   dependencies. Note that this only works because we make the
   assumption that one package does *not* overwrite files installed by
   another package.

   This is done for "extract dependencies" at the beginning of the
   extract step, and for "normal dependencies" at the beginning of the
   configure step.

This is basically enough to make per-package SDK and target work. The
only gotcha is that at the end of the build, output/target and
output/host are empty, which means that:

 - The filesystem image creation code cannot work.

 - We don't have a SDK to build code outside of Buildroot.

In order to fix this, this commit extends the target-finalize step so
that it starts by populating output/target and output/host by
rsync-ing into them the target and host directories of all packages
listed in the $(PACKAGES) variable. It is necessary to do this
sequentially in the target-finalize step and not in each
package. Doing it in package installation means that it can be done in
parallel. In that case, there is a chance that two rsyncs are creating
the same hardlink or directory at the same time, which makes one of
them fail.

This change to per-package directories has an impact on the RPATH
built into the host binaries, as those RPATH now point to various
per-package host directories, and no longer to the global host
directory. We do not try to rewrite such RPATHs during the build as
having such RPATHs is perfectly fine, but we still need to handle two
fallouts from this change:

 - The check-host-rpath script, which verifies at the end of each
   package installation that it has the appropriate RPATH, is modified
   to understand that a RPATH to $(PER_PACKAGE_DIR)/<pkg>/host/lib is
   a correct RPAT.

 - The fix-rpath script, which mungles the RPATH mainly for the SDK
   preparation, is modified to rewrite the RPATH to not point to
   per-package directories. Indeed the patchelf --make-rpath-relative
   call only works if the RPATH points to the ROOTDIR passed as
   argument, and this ROOTDIR is the global host directory. Rewriting
   the RPATH to not point to per-package host directories prior to
   this is an easy solution to this issue.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-11-29 14:24:05 +01:00
Thomas Petazzoni 085f32c17b package/pkg-generic.mk: make HOST_<pkg>_DL_OPTS inherit from <pkg>_DL_OPTS
Just like _SITE, _SOURCE, _SITE_METHOD, it is very likely that if
<pkg>_DL_OPTS is defined, the same value should be used for
HOST_<pkg>_DL_OPTS, so let's have the same inheritance logic than the
one we have for other variables.

Fixes:

  https://bugs.busybox.net/show_bug.cgi?id=12321

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-11-29 10:52:56 +01:00
Angelo Compagnucci 2ddaa31bef package/pkg-generic: fix space formatting
This patch fixes a formatting issues where spaces were used instead of tabs.

Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-11-25 18:19:59 +01:00
Yann E. MORIN 2c543b4f4c package/pkg-generic.mk: use site method for same-site extra downloads
When a package specifies extra downloads, it has the option to only name
the basename of the extra download, in which case that extra download
will be retrieved from the same location the main download is retrieved
from.

In that case, if the extra download contains a '+', it would confuse the
dl-wrapper, which believes the LHS of the '+' is the site method, and
the RHS the actual URI, and so the dl-wrapper mangles and damages the
URI when fetching such extra downloads, like that happens with android
tools, where the proper URI and mangled URIs of the extra download are,
respectively:

    https://launchpad.net/ubuntu/+archive/primary/+files/android-tools_4.2.2+git20130218-3ubuntu41.debian.tar.gz
    http://archive/primary/+files/android-tools_4.2.2+git20130218-3ubuntu41.debian.tar.gz

We fix that by always propagating the site method to extra downloads,
but only when they are specified as relative to the main download URI.

For the extra downloads that specify a full URI, it is not systematic
that it is the same site method. For example, a main download could be a
git clone, but an extra download a pure http download; in that case we
can't replicate the site method for extra downloads, so they'll have to
take appropriate care to specify the required method and encoding if
needed.

Reported-by: Jemy Zhang <jemy.zhang@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Jemy Zhang <jemy.zhang@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-11-08 21:52:51 +01:00
Yann E. MORIN 4e39730e44 core/legal-info: don't require overriding _LICENSE
Currently, the formatting we impose on the _LICENSE variable requires
that we also use the rarely used := assignment operator, which makes
the _LICENSE variable the only variable that users have to write with
this operator.

This really departs from the simplicity and consistency of using the
append-assignment, which we use for every other variable.

This is because the append-assignment operator surreptiously
introduces a space between the original value and the appended one. But
we can use this knowledge, to match any instance of a space followed by
a comma, and turn it into a single comma.

This allows users to now have a consistent use of the '=' and '+='
operators we use everywhere else in .mk files.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-27 10:35:06 +01:00
Michał Łyszczek 162044407b package/skeleton-init-openrc: add support for starting sysv scripts
Add an OpenRC service that starts and stops sysv-init scripts. We order
that script 'after local' so that it is started after all other native
openrc services.

Signed-off-by: Michał Łyszczek <michal.lyszczek@bofc.pl>
[yann.morin.1998@free.fr:
  - don't propagate the micro optimisation for running .sh scripts
  - use spaces, not TABs
  - stop services in reverse order
  - reword commit log
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-23 22:29:28 +02:00
Yann E. MORIN 9ee584c0dd core/pkg-infra: really build all dependencies of foo with foo-depends
Currently, foo-depends only builds build dependencies. This means that
download and extract dependencies are not built.

First, but a minor point, this is inconsistent with foo-show-depends,
which does display all dependencies:

    $ make host-gzip-show-depends
    host-skeleton host-tar
    $ make host-gzip-depends
        # Only host-skeleton is built and installed

Second, and more important, it makes it more difficult to preapre a
debug build, like so:

    $ make foo-depends
    $ tar cf output.tar output
    $ make foo
        # bummer, broken
        # edit foo.mk to try and fix it
    $ rm -rf output; tar xf output.tar
        # rince and repeat

Change foo-depends so that it really builds all the dependencies for
foo, bringing it on-par with foo-show-depends.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-07-03 23:04:50 +02:00
Jan Kundrát a27078d32d Allow overriding the VCS exclude list with *_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS
We have some internal packages which automatically bake a version string
from the git checkout, and we usually combine these with a
*_OVERRIDE_SRCDIR. I would like to let Buildroot *not* skip the .git
directory when picking up sources from the local checkout.  It turns out
that the existing mechanism (*_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS) only
supports adding to the exclude list because `rsync` simply uses the
first match from the provided filtering rules.

Solve this by using the user-provided values first. If they match, then
`rsync` won't exclude stuff based on the generic VCS exclude patterns.

Signed-off-by: Jan Kundrát <jan.kundrat@cesnet.cz>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-06-10 15:29:11 +02:00
Michał Łyszczek 6d89420774 package/pkg-generic.mk: add <pkg>_INSTALL_INIT_OPENRC
* package/pkg-generic.mk
  Add <pkg>_INSTALL_INIT_OPENRC so packages can define their own steps
  to install openrc service scripts.

* docs/manual/adding-packages-generic.txt
  update documentation about new hook.

Signed-off-by: Michał Łyszczek <michal.lyszczek@bofc.pl>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-05-18 23:21:19 +02:00
Yann E. MORIN f5f17c4dd7 core: remove show-dependency-tree
show-dependency-tree was introduced in this release cycle, as a way to
quickly and easily provide the dependency tree to graph-depends.

show-dependency-tree is no longer used, now that graph-depends has been
switched over to using the more versatile show-info.

Beside, show-dependency-tree has never been part of a release.

Drop it.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-05-07 23:06:05 +02:00
Yann E. MORIN 5abb88218e core: add per-package and per-filesystem show-info
Sometimes, it is need to quickly get the metadata of a subset of
packages, without resorting to a full-blown JSON query.

Introduce a new per-package (and per-filesystem) foo-show-info rule,
that otputs a per-entity valid JSON blob.

Note that calling it for multiple packages and.or filesystems at once
will not generate a valid JSON blob, as there would be no separator
between the JSON elements:

    $ make {foo,bar}-show-info
    { "foo": { foo stuff } }
    { "bar": { bar stuff } }

However, jq is able to absorb this, with its slurping ability, which
generates an array (ellipsed and manualy reformated for readability):

    $ make {foo,bar}-show-info |jq -s . -
    [
      { "foo": { foo stuff } },
      { "bar": { bar stuff } }
    ]

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-15 23:47:22 +02:00
Yann E. MORIN 5a0d681394 infra/pkg-download: make the DOWNLOAD macro fully parameterised
Currently, the DOWNLOAD macro is context-dependent and expects
the PKG variable to be set to the current package.

This is not so nice.

Change the macro to expect the upper-case package name as a
parameter, rather than expect it from a variable.

Adapt the caller accordingly.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-15 23:21:54 +02:00
Yann E. MORIN 254531acbb package/pkg-generic: simplify dependency tree
Why do things simply, when we can do it complicated?

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-04-03 22:55:45 +02:00
Norbert Lange 004960e967 package/pkg-generic: depend on host-{xz, lzip} only for fitting archives
Currently, host-xz and host-lzip are built as soon as the
corresponding tools are not provided by the system, independently of
whether they are really needed by the Buildroot configuration. This is
particularly annoying for host-lzip, which is only needed for very few
packages.

This commit modifies the generic package infrastructure to only add
host-lzip and host-xz as dependencies when really needed.

Signed-off-by: Norbert Lange <nolange79@gmail.com>
[Thomas:
 - improve commit log
 - as suggested by Yann E. Morin, make the lzip case similar to the xz
   case]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-04-01 22:57:17 +02:00
Yann E. MORIN ad89bffed5 package/pkg-generic: mark some rule PHONY
The %-graph-rdepends should be marked PHONY, as the %-graph-depends is.

The %-show-dependency-tree should also be a PHONY rule, because it does
not actually create a file. Furthermore, to avoid the "Nothing to be
done for .." message, give that rule an actual recipe that just does
nothing.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-25 19:27:04 +01:00
Yann E. MORIN 8623cc5deb package/pkg-generic: tweak only .la files that need it
Currently, when we tweak the .la files, we do so unconditionally on all
.la files, even those we already fixed in a previous run.

This has the nasty side effect that each .la file will be reported as
being touched by all packages that are installed after the package that
actually installed said .la file.

Since we can't easily know what files were installed by a package (that
is an instrumentation hook, and comes after the mangling), we use a
trick (like is done in libtool?): we do mangle all files, each into a
temporary location; if the result is identical to the input, we remove
the temporary, while if the result differs from the output, we move
the temporary to replace the input.

Reported-by: Nicolas Cavallari <nicolas.cavallari@green-communications.fr>
Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Nicolas Cavallari <nicolas.cavallari@green-communications.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-17 16:39:40 +01:00
Yann E. MORIN 870f37fe04 core: add make-based full-dependency list
Currently, when we need to build the full dependency graph, we call make
to show the list of packages (make show-targets), and then call it again
and again iteratively while it returns new packages.

Since calling make will parse the whole set of our Makefiles, this takes
quite a bit of time (~4s each here), and the total can get pretty long.

However, make being make, already builds the whole dependency tree
information, so we can just ask for it.

Add a new top-level rule 'show-dependency-tree' that displays the whole
set of dependencies for all packages. For each package, its name, type
and version is displayed, then all the direct, first-level dependencies
are dumped. We choose a format that is not unlike the dot-graph format,
because it is both easy to read as a human, and easy to parse as a
machine:

    foo: target 1.2.3
    foo -> bar host-meh
    bar: target virtual
    bar -> buz
    buz: target 2.3.4
    buz ->
    host-meh: host virtual
    host-meh -> host-bleark
    host-bleark: host 3.4.5
    host-bleark ->
    rootfs-meh: host
    rootfs-meh -> host-bleark

To be noted: rootfs are currently reported as if they were 'host'
packages, to stay aligned with how graph-depends currently treats them.
Ideally, graph-depends could be enhanced to recognise them separately,
but that is another story.

For just plain defconfig, which is about the smallest config we can have
with an internal toolchain, we already have a seven-fold improvement
(with the graph-depends rule modified to not run the pdf generation, to
be able to just compare the tree generation):

    $ time make graph-depends
    real    0m27.344s
    $ time make show-dependency-tree
    real    0m3.848s

>From defconfig, C++, wchar, locales, ssp, and allyespackageconfig,
tweaked for even more packages (qt5 not qt4, luajit to avoid multi
providers, etc...), the timings are (graph-depends still modified to
not generate the pdf):

    $ time make graph-depends
    real    1m56.459s
    $ time make show-dependency-tree
    real    0m5.748s

There. I don't think those numbers need any explanation whatsoever;
they do speak on their own. OK, for maths sake, the ratio is about
twenty-fold. So, "yeah", I guess... ;-)

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-17 15:39:12 +01:00
Yann E. MORIN 29aeb02aa0 infra/pkg-generic: use pure Makefile-based recursive dependencies
Calling to the graph-depends script is very costly, as it calls back to
'make' a lot of time.

It turns out that we already have the list of recursive dependencies, so
we can just print that.

As for the recursive reverse dependencies, we use the same memoisation
technique to cut-down on the expansion cost, which would otherwise be on
the order of 𝑶(𝑛²) (with 𝑛 enabled packages).

>From a defconfig, modified to use glibc, C++, wchar, locales, ssp, and
allyespackageconfig (tweaked to avoid multi providers, etc...), the
timings for X-show-recursive-rdepends are:

                    before      after       speedup     #rdeps
    libnss          0m22.932s   0m5.775s     3.97x      3
    qt5base         0m41.176s   0m5.781s     7.12x      67
    libjpeg         0m56.185s   0m5.749s     9.71x      228
    libxml2         0m54.964s   0m5.795s     9.48x      271
    freetype        0m46.754s   0m5.819s     8.07x      287
    libpng          0m53.577s   0m5.760s     9.30x      303
    sqlite          1m15.222s   0m5.807s    12.95x      801
    libopenssl      1m25.471s   0m5.844s    14.63x      931
    readline        1m13.805s   0m5.775s    12.78x      958
    libzlib         1m11.807s   0m5.820s    12.34x      1039
    toolchain       1m23.712s   0m6.080s    13.77x      2107
    skeleton        1m27.839s   0m6.293s    13.96x      2111 (+1)
    host-skeleton   1m27.405s   0m6.350s    13.76x      2172 (+2)

  - speedup: ratio before/after
  - #rdeps: number of recursive reverse dependencies, with the extra
            dependencies returned with this patch, see below for the
            reason.

So, for a low-level package with a lot of reverse dependencies, like
libzlibz, libopenssl or readline are, the timings are already very much
in favour of the change. This is less impressive with packages that
have few dependencies (libnss), but still much faster.

Also, remember that the config tested has as much packages enabled as
possible, so is in itself a degenerate case. With simpler and more
realistic configurations, the gains would probably be a bit lower than
reported above, but various tests still report good improvements
overall (note: coming up with a 'realistic' configuration is pretty
hard, as everyone and their dog have their notion of what is realistic
in their context, so nothing displayed here; timings are left as an
exercise for the interested parties to report aggravation in their
cases should they notice some regression).

Note that, more recursive reverse dependencies may be displayed now,
since we do not apply the exceptions applied in graph-depends. For
example, host-skeleton gains two new recursive reverse dependencies:
skeleton and toolchain, which are both exceptions in graph-depends.

As for direct (not reverse) dependencies: the gain is not as fantastic
as for reverse ones, but it is still noticeable, especially thanks to
a21212fb7c (package/pkg-generic: speed up RECURSIVE_FINAL_DEPENDENCIES);
just a few examples for %-show-recursive-depends:

                    before      after       speedup     #deps
    libzlib         0m46.864s   0m5.902s     7.94x      17
    qt5base         0m57.590s   0m5.848s     9.85x      190
    sqlite          0m46.601s   0m5.816s     8.01x      24

Basically, displaying recursive dependencies, direct or reverse, is
almost a constant now: it only slightly varies by about 10% depending
on the complexity of the dependency chain, with the parsing of the
Makefiles still accounting for the large majority of the time.

(PS. Thanks to Joseph for suggesting a list of interesting packages
to test, and thanks to Trent for his example of memoisation!)

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Joseph Kogut <joseph.kogut@gmail.com>
Cc: Trent Piepho <tpiepho@impinj.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-17 15:02:01 +01:00
Trent Piepho a21212fb7c package/pkg-generic: speed up RECURSIVE_FINAL_DEPENDENCIES
Evaluating all the <PKG>_RECURSIVE_FINAL_DEPENDENCIES variables
(abbreviated RFD hereafter) ends up being quite slow.  Enough, on a
reasonable modern workstation, to increase the time it takes to run
"make printvars" from 13 seconds in 2018.02 to 371 seconds in 2019.02.

This patch improves this by using dynamic programming to speed the
evaluation of RFD, reducing the before mentioned printvars time to about
14.6 seconds.

The evaluation of PKG1_RFD requires recursively evaluating each of
PKG1's dependencies' RFDs, then their dependencies' RFDs, and so on.
The same is done for PKG2_RFD.  But it's likely that many of the
dependencies of PKG2 are the same as PKG1.  And when we consider all
packages, the dependencies are re-computed many thousands of times.

To avoid this re-computation we memoize, or save, the computed value of
each RFD variable when it found the first time.  Subsequent evaluations
re-use the memoized value.

Surprisingly, this ends up being not all the hard to implement in make.
The basic construct is this:

VAR = $(if !defined(VAR__X),$(eval VAR__X := value))$(VAR__X)

The first time VAR is evaluated VAR__X will not be defined, and code to
set VAR__X to the computed value is eval'd.  Then the now defined value
of VAR__X is returned.  Subsequent evaluations can just return VAR__X.

It is important to note that VAR is defined with '=', as not enough
information (namely, all packages' dependencies) is know when it is
parsed to find the correct value.  VAR will be evaluated each time it is
used.  But VAR__X is defined with ":=", so that it is evaluated once
when defined, and not each time it is used.

Signed-off-by: Trent Piepho <tpiepho@impinj.com>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-03-01 11:12:38 +01:00
Yann E. MORIN 3c8f0d9efa core/pkg-infra: restore completeness of packages files lists
In commit 7fb6e78254 (core/instrumentation: shave minutes off the
build time), the built stampfile is used as a reference to detect files
installed by a package.

However, packages may install files keeping their mtime intact, and we
end up not detecting this. For example, the internal skeleton package
will install (e.g.) /etc/passwd with an mtime of when the file was
created in $(TOP_DIR), which could be the time the git repository was
checked out; that mtime is always older than the build stamp file, so
files installed by the skeleton package are never accounted for to that
package, or to any other package for that matters.

We switch to an alternate solution, which consists of storing some extra
metadata per file, so that we can more reasily detect modifications to
the files. Then we compare the state before the package is installed (by
reusing the existing list) and after the package is installed, compare
that to list any new file or modified files (in reality, ignoring
untouched and removed files). Finally, we store the file->package
association in the global list and store the new stat list as the global
list.

The format used for the .stat file is:

mtime:inode:perms:filetype:size,filename

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Trent Piepho <tpiepho@impinj.com>

[Peter: rename files, reformat, only look for files and symlinks and pass
	LC_ALL=C to comm as pointed out by Thomas De Schampheleire]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-08 09:40:53 +01:00
Yann E. MORIN ba015cdf04 package/pkg-generic: use readlink instead of realpath
realpath is missing on oldish distributions, like Debian 7, which is
still used in the wild.

Use readlink instead; that has been available since the dawn of ages now
(well, coreutils had it in 2003).

Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-02 11:22:11 +01:00
Peter Korsgaard 13c43455a0 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-12-02 08:16:10 +01:00
Yann E. MORIN 9c43f28de4 package/pkg-generic: ensure absolute paths in config-script fixups
In case a config script is called from a relative path, the $(dirname
$0) would return a relative path too.

Those paths are usually parts of includes or libraries search
directories, and the packagfes buildsystems may chdir() anywhere, and
thus the relative path will no longer be valid. For example:

  $ ./host/powerpc-buildroot-linux-uclibc/sysroot/usr/bin/net-snmp-config --cflags
  [...] -I./host/powerpc-buildroot-linux-uclibc/sysroot/usr/bin/../../.././bin/../powerpc-buildroot-linux-uclibc/sysroot/usr/include/libnl3 [...]

Canonicalise the path to be sure we use absolute paths.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-29 23:00:26 +01:00
Thomas Petazzoni a18c828bed package/pkg-generic: adjust config scripts tweaks for per-package directories
This commit adjusts the logic in pkg-generic.mk that tweaks the
*-config shell scripts installed by various libraries to make it
compatible with per-package directories.

This requires two fixes:

 - replacing $(STAGING_DIR) with a relative path from the config script
   to the staging directory, rather than using an absolute path of the
   staging directory.

   Without this, a *-config script provided by package A, but called
   from package B per-package directory will return paths from package A
   per-package directory:

   $ ./output/per-package/mcrypt/host/usr/<tuple>/sysroot/usr/bin/libmcrypt-config --libs
   -L..../output/per-package/libmcrypt/host/usr/<tuple>/sysroot/usr/lib/

   The libmcrypt-config script is installed by the libmcrypt package,
   and mcrypt is a package that depends on libmcrypt. When we call the
   libmcrypt-config script from the mcrypt per-package directory, it
   returns a -L flag that points to the libmcrypt per-package
   directory.

   One might say: but this is OK, since the sysroot of the libmcrypt
   per-package directory also contains the libmcrypt library. This is
   true, but we encounter a more subtle issue: because -L paths are
   considered before standard paths, ld ends up finding libc.so in the
   libmcrypt per-package directory. This libc.so file is a linker
   script that looks like this:

   GROUP ( /lib/libc.so.6 /usr/lib/libc_nonshared.a  AS_NEEDED ( /lib/ld-linux.so.3 ) )

   Normally, thanks to ld sysroot awareness, /lib/libc.so.6 in this
   script is re-interpreted according to the sysroot. But in this
   case, the library is *outside* the compiler sysroot. Remember: we
   are using the compiler/linker from the "mcrypt" per-package
   directory, but we found "libc.so.6" in the "libmcrypt" per-package
   directory.

   This causes the linker to really use the /lib/libc.so.6 from the
   host machine, obvisouly leading to a build failure such as:

   output/per-package/libgcrypt/host/opt/ext-toolchain/bin/../lib/gcc/nios2-linux-gnu/7.3.1/../../../../nios2-linux-gnu/bin/ld: cannot find /lib/libc.so.6
   output/per-package/libgcrypt/host/opt/ext-toolchain/bin/../lib/gcc/nios2-linux-gnu/7.3.1/../../../../nios2-linux-gnu/bin/ld: cannot find /usr/lib/libc_nonshared.a
   output/per-package/libgcrypt/host/opt/ext-toolchain/bin/../lib/gcc/nios2-linux-gnu/7.3.1/../../../../nios2-linux-gnu/bin/ld: cannot find /lib/ld-linux-nios2.so.1

 - Some *-config scripts, such as the apr-1-config script, contain
   references to host tools:

   CC=".../output/per-package/apr/hosr/bin/arm-linux-gcc"
   CCP=".../output/per-package/apr/hosr/bin/arm-linux-cpp"

   We also want to replace those with proper relative paths. To
   achieve this, we need to also replace $(HOST_DIR) with a relative
   path. Since $(STAGING_DIR) is inside $(HOST_DIR), the first
   replacement of $(STAGING_DIR) by @STAGING_DIR@ is no longer needed:
   replacing $(HOST_DIR) by @HOST_DIR@ is sufficient. We still need to
   replace @STAGING_DIR@ by the proper path though, as we introduce
   @STAGING_DIR@ references in exec_prefix and prefix variables, as
   well as -I and -L flags.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-11-26 19:12:18 +01:00
Thomas Petazzoni d0f4f95e39 Makefile: rework main directory creation logic
In the current code, the creation of the main output directories
(BUILD_DIR, STAGING_DIR, HOST_DIR, TARGET_DIR, etc.) is done by a
global "dirs" target. While this works fine in the current situation,
it doesn't work well in a context where per-package host and target
directories are used.

For example, with the current code and per-package host directories,
the output/staging symbolic link ends up being created as a link to
the per-package package sysroot directory of the first package being
built, instead of the global sysroot.

This commit reworks the creation of those directories by having the
package/pkg-generic.mk code ensure that the build directory, target
directory, host directory, staging directory and binaries directory
exist before they are needed.

Two new targets, host-finalize and staging-finalize are added in the
main Makefile to create the compatibility symlinks for host and
staging directories. They will be extended later with additional logic
for per-package directories.

Thanks to those changes, the global "dirs" target is entirely removed.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-11-26 19:09:46 +01:00
Yann E. MORIN 2218dc85be support/dependencies: add a check for a suitable gzip
Recently, some hash mismatch have been reported, both by users as well
as autobuilder failures, about tarballs generated from git repositories.

This turned out to be caused by users having the 'gzip' command somehow
aliased to 'pigz' (which stand for: parallel implementation of gzip,
which takes advantage of multi-processor system to parallelise the
compression).

Unfortunately, the output of pigz-compressed archives differ from that
of gzip (even though they *are* valid gzip-compressed streams).

Add a dependency check that ensures that gzip is not pigz. If that is
the case, define a conditional dependency to host-gzip, that is used as
a download dependency for packages that will generate compressed files,
i.e. cvs, git, and svn.

Fixes:
    http://autobuild.buildroot.org/results/330/3308271fc641cadb59dbf1b5ee529a84f79e6d5c/

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Marcin Niestrój <m.niestroj@grinn-global.com>
Cc: Erico Nunes <nunes.erico@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-24 15:14:58 +01:00
Serj Kalichev 75c81a12f6 package/pkg-generic.mk: fix show-build-order stdout pollution
The commands like "make show-build-order" or "make
<package>-show-build-order" show the build order and then print
"make[1]: Nothing to be done for 'show-build-order'" to stdout. It
pollutes output. Technically this message is true but it's not true
for user because he gets an information.

The <package>-show-build-order targets use $(info) for package name
printing.  The make utility doesn't consider the internal directive as
a command so it think that it's "Nothing to be done". The patch adds
the empty command to <package>-show-build-order to inform make utility
that taget makes some real actions.

Signed-off-by: Serj Kalichev <serj.kalichev@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Thomas: invert $(info) and @:, as suggested by Yann.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-19 22:25:30 +01:00
Yann E. MORIN 5b5feea875 package/pkg-generic: add host-tar dependency for downloads from repositories
Three of our download backends need a host tar that can generate
reproducible archives: cvs, git, and svn. The other two, bzr and hg,
use their internal implementation.

So, for those three that need it, and a dependency on host-tar when the
system tar is not appropriate.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-01 22:01:12 +01:00
Yann E. MORIN 60d8633ea3 package/pkg-genric: add possibility to declare download dependencies
For some packages, we may need to have a certain set of host-tools built
before the download of said packages are attempted. For example, when
the system tar is not suitable, we will want to build our own tar before
we attempt a git download (because we generate a tarball in the git
backend).

Mimick the _EXTRACT_DEPENDENCIES, and introduce _DOWNLOAD_DEPENDENCIES.
As for _EXTRACT_DEPENDENCIES, we do not document _DOWNLOAD_DEPENDENCIES,
on the assumption that it is mostly for internal use.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-01 21:35:30 +01:00
Yann E. MORIN e8c0caadec package/pkg-generic: postpone evaluation of dependency conditions
In the pkg-inner macros, all variables, but the positional arguments,
must be $$-prefixed, so that they are expanded only when the macro is
evaluated in each package, not when the macro is parsed.

It is to be noted, though, that the current code, even though
incorrect by the above rules, seemed to work. However, the upcoming
addition of download dependencies, mimicking that code, would not work
unless it was $$-prefixed.

So, for consistency sake, and for correctness sake, let's always use
the $$-prefix in the inner macro.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-01 21:31:36 +01:00
Michal Sojka e33ea1c9a2 core/legal-info: Add package dependencies with licenses to the manifest
This adds one column to the legal-info manifest table. It contains the
dependencies of the given package and their licenses. This information
is useful when assessing license compatibility of the packages and
their libraries.

An example of the content of the new column for the MPD package is
shown below:

    "alsa-lib [LGPL-2.1+ (library), GPL-2.0+ (aserver)] boost
    [BSL-1.0] libid3tag [GPL-2.0+] libmad [GPL-2.0+] libogg
    [BSD-3-Clause] libvorbis [BSD-3-Clause] libzlib [Zlib]
    skeleton-init-common [unknown] skeleton-init-sysv [unknown] sqlite
    [Public domain] toolchain-external-linaro-arm [unknown]"

[Credits to Yann E. MORIN <yann.morin.1998@free.fr> for suggesting a
few simplifications.]

Signed-off-by: Michal Sojka <sojka@merica.cz>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-10-21 19:13:20 +02:00
Michal Sojka 8f14901043 core/legal-info: Change order of legal-manifest parameters
The last parameter {HOST|TARGET} is now first. With this change,
adding new columns to the legal manifest file (as in the next commit)
will be slightly easier to review.

Signed-off-by: Michal Sojka <sojka@merica.cz>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-10-21 19:12:31 +02:00