Commit graph

1172 commits

Author SHA1 Message Date
Yegor Yefremov ea31dc1cd4 support/run-tests: check for empty sequences in a pythonic way
According to PEP8 empty sequences should be checked as booleans.

Fixes the following PEP8 warning:
Do not use `len(SEQUENCE)` to determine if a sequence is empty

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-16 15:38:03 +01:00
Thomas Petazzoni 4a157be9ef support/scripts/pkg-stats: add support for CVE reporting
This commit extends the pkg-stats script to grab information about the
CVEs affecting the Buildroot packages.

To do so, it downloads the NVD database from
https://nvd.nist.gov/vuln/data-feeds in JSON format, and processes the
JSON file to determine which of our packages is affected by which
CVE. The information is then displayed in both the HTML output and the
JSON output of pkg-stats.

To use this feature, you have to pass the new --nvd-path option,
pointing to a writable directory where pkg-stats will store the NVD
database. If the local database is less than 24 hours old, it will not
re-download it. If it is more than 24 hours old, it will re-download
only the files that have really been updated by upstream NVD.

Packages can use the newly introduced <pkg>_IGNORE_CVES variable to
tell pkg-stats that some CVEs should be ignored: it can be because a
patch we have is fixing the CVE, or because the CVE doesn't apply in
our case.

>From an implementation point of view:

 - A new class CVE implement most of the required functionalities:
   - Downloading the yearly NVD files
   - Reading and extracting relevant data from these files
   - Matching Packages against a CVE

 - The statistics are extended with the total number of CVEs, and the
   total number of packages that have at least one CVE pending.

 - The HTML output is extended with these new details. There are no
   changes to the code generating the JSON output because the existing
   code is smart enough to automatically expose the new information.

This development is a collective effort with Titouan Christophe
<titouan.christophe@railnova.eu> and Thomas De Schampheleire
<thomas.de_schampheleire@nokia.com>.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-15 16:49:07 +01:00
Romain Naour faec5c583e support/testing/glxinfo: explicitely enable GLX
Since [1], the GLX support is enabled by BR2_PACKAGE_MESA3D_OPENGL_GLX
symbol.

Since [2], only one swrast provider can be built.
Keep BR2_PACKAGE_MESA3D_DRI_DRIVER_SWRAST.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/400391349

[1] 5cb821d563
[2] 09a0a28507

Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-15 11:59:13 +01:00
Peter Korsgaard 5fd8dd203a toolchain: use consistent code style for C code
Most, but not all our C code follows the Linux kernel code style (as
documented in Documentation/process/coding-style.rst).  Adjust the few
places doing differently:

- Braces:
  ..but the preferred way, as shown to us by the prophets Kernighan
  and Ritchie, is to put the opening brace last on the line

- Spaces after keywords:
  Use a space after (most) keywords

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-02-08 22:10:06 +01:00
Vincent Fazio 338e62bd5d toolchain: allow using custom headers newer than latest known ones
When Buildroot is released, it knows up to a certain kernel header
version, and no later. However, it is possible that an external
toolchain will be used, that uses headers newer than the latest version
Buildroot knows about.

This may also happen when testing a development, an rc-class, or a newly
released kernel, either in an external toolchain, or with an internal
toolchain with custom headers (same-as-kernel, custom version, custom
git, custom tarball).

In the current state, Buildroot would refuse to use such toolchains,
because the test is for strict equality.

We'd like to make that situation possible, but we also want the user not
to be lenient at the same time, and select the right headers version
when it is known.

So, we add a new Kconfig blind option that the latest kernel headers
version selects. This options is then used to decide whether we do a
strict or loose check of the kernel headers.

Suggested-by: Aaron Sierra <asierra@xes-inc.com>
Signed-off-by: Vincent Fazio <vfazio@xes-inc.com>
[yann.morin.1998@free.fr:
  - only do a loose check for the latest version
  - expand commit log
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Vincent Fazio <vfazio@xes-inc.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-02-08 20:25:10 +01:00
Titouan Christophe 1b2319f931 support/testing: add runtime test for Crudini (py2 and py3)
This also adds the new tests to the gitlab CI configuration.

Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-02-05 16:23:42 +01:00
Matt Weber 496a43c777 support/testing: new opkg test case
- Validates an archive can be installed and removed
- Builds an archives that uses postinst and prerm scripts

Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-02-03 22:17:42 +01:00
Yegor Yefremov 96bbea75a3 support/testing: add python-can test case
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2020-01-31 08:05:21 +01:00
Luca Ceresoli ec78068972 core/legal-info: update list of saved material in README
The README file saved by legal-info does not mention the host package
variant of the saved material. Add them.

Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-01-18 18:38:42 +01:00
Yann E. MORIN a95ea4b64d core/dependencies: check if we need to build our own host-coreutils
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-01-06 21:43:38 +01:00
Yann E. MORIN ed56b7b547 support/testing: really fix python-gitdb2 test
The test-case for python-gitdb2 consists solely in verifying that the
module can indeed be imported.

However, flake8 errors out on unused imports. Furthermore, it also
errors about wildcard imports, as it can detect unused symbols.

Commit d8c86be9cd (support/testing: fix python-gitdb2 test) tried to
address this issue, by explicitly squelching the two errors, F401 and
F403.

While that works on recent distros, the image used by our docker
pipeline is laggign behind and the flake8 there only handles at most a
single error in the noqa list.

Do as is done with the other python samples, and just blindly ignore
all errors.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2020-01-04 09:29:36 +01:00
Yann E. MORIN 8dbea2216d support/config-fragments/autobuild: fix riscv toolchains
In commit aee39cbf27 (arch/riscv: set the default float ABI based on
ISA extensions), the default ABI changed, so the config fragments used
by the autobuilders were adapated accordingly, in commit f89871e810
(support/config-fragments: fix br-riscv{32,64} toolchain fragments).

But now, we need to revert again, because the newer toolchains are now
using the default ABI again.

We do not really do a revert, though, because the original change was
right, and a revert would mean it was not.

Fixes:
    http://autobuild.buildroot.org/results/b59/b593267fb9fc9a002b977e049b2a5389dbaded30/ (riscv32)
    http://autobuild.buildroot.org/results/b42/b42a4b22b29f47d5c85be119b310f1dfb61112a1/ (riscv64)
    ... and so many others on various packages...

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Mark Corbin <mark.corbin@embecosm.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-01-02 09:15:43 +01:00
Thomas Petazzoni 2d67b66df9 support/config-fragments/autobuild: update Buildroot toolchains to 2019.11
Simple bump of the toolchain components. For nios2, the toolchain now
has SSP support as well.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
  - fix version it commit title
  - mention SSP for nios2
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2019-12-30 13:18:03 +01:00
Heiko Thiery 89f5e98932 support/download/svn: generate reproducible svn archives
To generate a reproducible archive from a svn repository mainly the same
aproach is done like for the archives from a git repository.

Signed-off-by: Heiko Thiery <heiko.thiery@gmail.com>
[yann.morin.1998@free.fr: get the date of the revision]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2019-12-30 10:50:21 +01:00
Titouan Christophe 9fa2add810 support/testing: add test for python-avro
This adds a test case for python-avro, with a script that
performs a simple deserialization.

Signed-off-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2019-12-29 21:38:33 +01:00
Yann E. MORIN d8c86be9cd support/testing: fix python-gitdb2 test
Cthe test-case for python-gitdb2 consists solely in verifying that the
module can indeed be imported.

However, flake8 errors out on unused imports. Furthermore, it also
errors about wildcard imports, as it can detect unused symbols.

Squelch those errors.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2019-12-18 21:12:22 +01:00
Thomas Petazzoni 2e42840e2a package/pkg-utils.mk: rework implementation of extractor-system-dependency
Now that we have the EXTRACTOR_PKG_DEPENDENCY.* variables available,
we can use them to implement extractor-system-dependency: if for a
given archive type, the corresponding EXTRACTOR_PKG_DEPENDENCY.<type>
variable is empty, then it means we need the corresponding extractor
tool to be provided by the system.

Following this, EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS is no
longer used, so we can drop it from support/dependencies/.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2019-12-17 22:09:48 +01:00
Adam Duskett 5ef1679635 support/testing: add gitdb2 test
Add a simple test case that imports the module.

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-12-16 23:36:45 +01:00
Adam Duskett f81f673a78 support/testing: add smmap2 test
Add a simple test case that imports the module and instantiates a new
SlidingWindowMapManager class.

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-12-16 23:28:08 +01:00
Thomas Petazzoni fbee5c8e91 Revert "support/download/git: rename local refs to avoid confusing Git warning"
This reverts commit 6f35d96756.

Repeat after me: on the master branch you will not work. On the master
branch you will not work.

This definitely shouldn't have been pushed. Sorry about that.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-12-12 21:24:35 +01:00
Thomas Petazzoni 6f35d96756 support/download/git: rename local refs to avoid confusing Git warning
Running "git fetch origin ${cset}:${cset}" to create a local ref
${cset} from the remote ref ${cset} causes Git to issue a warning like
the below, when the version is a full commit hash:

===

warning: refname '49eb4ecb1ef9879ebc6789a1bdb536ab2b1d9871' is ambiguous.
Git normally never creates a ref that ends with 40 hex characters
because it will be ignored when you just specify 40-hex. These refs
may be created by mistake. For example,

  git switch -c $br $(git rev-parse ...)

where "$br" is somehow empty and a 40-hex ref is created. Please
examine these refs and maybe delete them. Turn this message off by
running "git config advice.objectNameWarning false"

===

This warning is very confusing for users, and is caused by the fact
that Git doesn't like our local ref name to look like a commit hash.

So, this commit proposes to fix the issue by having the local ref
named buildroot-${cset}, i.e
buildroot-${version-specified-by-the-package}.

The generated tarballs are exactly identical, nothing changes, it is
really just internally the local ref we are using to checkout the
correct version that is different. And it avoids the confusing
warning.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-12-12 15:42:50 +01:00
Thomas Petazzoni 97dee44a6c package/pkg-generic.mk, support/scripts/fix-rpath: fix per-package regexp
Commit c4e6d5c8be ("core: implement
per-package SDK and target") had a mistake on the regexp that is used
to match $(PER_PACKAGE_DIR)/<something>/, and due to this, the regexp
was never matched.

The + sign in [^/]+ which was suggested by Yann E. Morin during the
review of the per-package patch series (instead of [^/]*) needs to be
escaped to be taken into account correctly. Without this, the regexp
doesn't match, and the replacement is not done, causing:

 (1) For the libtool fixup in pkg-generic.mk, the lack of replacement
     causes libtool .la files to not be tweaked as expected, which it
     turn causes build failures reported by the autobuilder.

 (2) For the fix-rpath, the RPATH of host binaries in the SDK were not
     correct.

Interestingly, we have the same regexp in
support/scripts/check-host-rpath, but here the + sign does not need to
be escaped.

Fixes:

  http://autobuild.buildroot.net/results/d4d996f3923699e266afd40cc7180de0f7257d99/ (libsvg-cairo)
  http://autobuild.buildroot.net/results/56330f86872f67a2ce328e09b4c7b12aa835a432/ (bind)
  http://autobuild.buildroot.net/results/9e0fc42d2c9f856b92954b08019b83ce668ef289/ (ibrcommon)
  and probably a number of other similar issues

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-12-12 08:27:54 +01:00
Arnout Vandecappelle (Essensium/Mind) 17638bc3ad support/testing: test_python_django: add missing line
flake8 complains with:

test_python_django.py:25:1: E302 expected 2 blank lines, found 1

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-12-06 00:32:34 +01:00
Adam Duskett 8583b2c6d8 support/testing: add python-django test
This test comprises of four simple steps:
  1: Start a new simple project called testsite.
  2: Run ./manage.py migrate on the new testsite.
  3: Run ./manage.py runserver 0.0.0.0:1234 & sleep 30
    - The sleep 30 is necessary as it may take several seconds for
      the django server to fully start.
  4: Run netstat to ensure the server opened port 1234.

Signed-off-by: Adam Duskett <aduskett@greenlots.com>
[Thomas: use self.assertRunOk() when appropriate]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-12-05 23:02:14 +01:00
Adam Duskett 600132f8c0 support/testing: add python-gobject test
Add a simple test case that imports the module.

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-12-03 23:49:34 +01:00
Peter Korsgaard 7b24bd59c8 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-12-02 09:39:41 +01:00
Peter Korsgaard 836b84a774 Update for 2019.11
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-12-01 22:39:47 +01:00
Thomas Petazzoni c4e6d5c8be core: implement per-package SDK and target
This commit implements the core of the move to per-package SDK and
target directories. The main idea is that instead of having a global
output/host and output/target in which all packages install files, we
switch to per-package host and target directories, that only contain
their explicit dependencies.

There are two main benefits:

 - Packages will now see only the dependencies they explicitly list in
   their <pkg>_DEPENDENCIES variable, and the recursive dependencies
   thereof.

 - We can support top-level parallel build properly, because a package
   only "sees" its own host directory and target directory, isolated
   from the build of other packages that can happen in parallel.

It works as follows:

 - A new output/per-package/ directory is created, which will contain
   one sub-directory per package, and inside it, a "host" directory
   and a "target" directory:

   output/per-package/busybox/target
   output/per-package/busybox/host
   output/per-package/host-fakeroot/target
   output/per-package/host-fakeroot/host

   This output/per-package/ directory is PER_PACKAGE_DIR.

 - The global TARGET_DIR and HOST_DIR variable now automatically point
   to the per-package directory when PKG is defined. So whenever a
   package references $(HOST_DIR) or $(TARGET_DIR) in its build
   process, it effectively references the per-package host/target
   directories. Note that STAGING_DIR is a sub-dir of HOST_DIR, so it
   is handled as well.

 - Of course, packages have dependencies, so those dependencies must
   be installed in the per-package host and target directories. To do
   so, we simply rsync (using hard links to save space and time) the
   host and target directories of the direct dependencies of the
   package to the current package host and target directories.

   We only need to take care of direct dependencies (and not
   recursively all dependencies), because we accumulate into those
   per-package host and target directories the files installed by the
   dependencies. Note that this only works because we make the
   assumption that one package does *not* overwrite files installed by
   another package.

   This is done for "extract dependencies" at the beginning of the
   extract step, and for "normal dependencies" at the beginning of the
   configure step.

This is basically enough to make per-package SDK and target work. The
only gotcha is that at the end of the build, output/target and
output/host are empty, which means that:

 - The filesystem image creation code cannot work.

 - We don't have a SDK to build code outside of Buildroot.

In order to fix this, this commit extends the target-finalize step so
that it starts by populating output/target and output/host by
rsync-ing into them the target and host directories of all packages
listed in the $(PACKAGES) variable. It is necessary to do this
sequentially in the target-finalize step and not in each
package. Doing it in package installation means that it can be done in
parallel. In that case, there is a chance that two rsyncs are creating
the same hardlink or directory at the same time, which makes one of
them fail.

This change to per-package directories has an impact on the RPATH
built into the host binaries, as those RPATH now point to various
per-package host directories, and no longer to the global host
directory. We do not try to rewrite such RPATHs during the build as
having such RPATHs is perfectly fine, but we still need to handle two
fallouts from this change:

 - The check-host-rpath script, which verifies at the end of each
   package installation that it has the appropriate RPATH, is modified
   to understand that a RPATH to $(PER_PACKAGE_DIR)/<pkg>/host/lib is
   a correct RPAT.

 - The fix-rpath script, which mungles the RPATH mainly for the SDK
   preparation, is modified to rewrite the RPATH to not point to
   per-package directories. Indeed the patchelf --make-rpath-relative
   call only works if the RPATH points to the ROOTDIR passed as
   argument, and this ROOTDIR is the global host directory. Rewriting
   the RPATH to not point to per-package host directories prior to
   this is an easy solution to this issue.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-11-29 14:24:05 +01:00
Pierre-Jean Texier bda4008975 support/testing: basetest.py: fix code style
Fix these warnings:

W291 trailing whitespace

Fixes:
 - https://gitlab.com/buildroot.org/buildroot/-/jobs/360824861

Signed-off-by: Pierre-Jean Texier <pjtexier@koncepto.io>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-11-25 21:40:52 +01:00
Pierre-Jean Texier bd18e74eeb support/testing: test_lxc.py: fix code style
Fix these warnings:

E122 continuation line missing indentation or outdented
E127 continuation line over-indented for visual indent
E265 block comment should start with '# '
E302 expected 2 blank lines, found 1
F401 'pexpect' imported but unused

Fixes:
 - https://gitlab.com/buildroot.org/buildroot/-/jobs/360824861

Signed-off-by: Pierre-Jean Texier <pjtexier@koncepto.io>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-11-25 21:40:41 +01:00
Patrick Havelange 1ca16b1feb support/testing: add lxc test
The test starts a simple container with an iperf3 server.
The container is using the tini init system, with a shared rootfs.
An iperf3 client is started from the host to check that the container
is really up and running.

Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-11-20 22:38:25 +01:00
Patrick Havelange 63966e56a3 support/testing: add assertRunOk method to BRTest class
This method asserts that the given command ran successfully.
The goal is for it to be used by the different tests when needed.

Signed-off-by: Patrick Havelange <patrick.havelange@essensium.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-11-20 22:34:53 +01:00
James Hilliard d4773b610a support/dependencies: set cmake version min to 3.10
This is required by wpewebkit and webkitgtk.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Reviewed-by: Adrian Perez de Castro <aperez@igalia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-28 23:01:47 +01:00
Ricardo Martincoski 4a40d36f13 support/testing: switch to Python 3 only
Python 2.7 will not be maintained past 2020.

Many scripts on the tree are used during the build and should keep
Python 2 compatibility for a while.
This is not the case for the runtime test infra. It's meant to be run in
modern distros only, so it can safely switch to support Python 3 only.

An advantage of this approach is to have less scenarios to test in.
Otherwise every change to the test infra or runtime tests would need to
be tested against both versions of the interpreter, increasing the
effort of the developers, to ensure the compatibility to Python 2 was
not broken.

In order to accomplish the change to Python 3:
 - change the shebang for run-tests;
 - use Python 3 urllib as a drop-in replacement for Python 2 urllib2;
 - when writing the downloaded binary files, explicitly open the output
   file as binary;
 - when subprocess is used to retrieve the text output from commands,
   explicitly ask for text output. For this, use 'universal_newlines'
   because 'text' was added only on Python 3.7;
 - when pexpect is used to retrieve the text output from qemu or git,
   explicitly ask for text output using 'encoding';
 - the code using csv currently follows the example in the documentation
   for the Python 2 module, change it to follow the example in the
   documentation for the Python 3 module;
 - fix the relative import for test_git.py to be Python 3 compliant.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Tested-by: Romain Naour <romain.naour@smile.fr>
Tested-by: Nicolas Carrier <nicolas.carrier@orolia.com>
Signed-off-by: Nicolas Carrier <nicolas.carrier@orolia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-10-28 22:14:07 +01:00
Ricardo Martincoski 8b44338ec5 support/docker: add python3
The test infra will soon be converted to Python 3 only.
So add the interpreter and also the Python 3 variant of modules nose2
and pexpect to the docker image used to run runtime tests.

Keep the Python 2 variant of those modules to allow a gradual
transition.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Nicolas Carrier <nicolas.carrier@orolia.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-27 20:24:10 +01:00
Carlos Santos 31d1fb27b0 support/scripts/genimage.sh: pass an empty rootpath to genimage
genimage makes a full copy of the given rootpath to ${GENIMAGE_TMP}/root
so passing TARGET_DIR would be a waste of time and disk space. We don't
rely on genimage to build the rootfs image, just to insert a pre-built
one in the disk image.

Signed-off-by: Carlos Santos <unixmania@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-27 12:19:32 +01:00
Yann E. MORIN 5ee815ab6a docs/manual: bump our gcc/g++ requirements to 4.8
Currently, we only require a gcc 4.4 version, which now is pretty old
(released in April 2009).  This requirement is not even tested nowadays,
with our oldest autobuilder having a 4.7 version only.

And even then, 4.7 is still old enough that it prevents us from
upgrading some packages. For example cmake 3.10+ requires C++11
constructs that were only added in gcc 4.8 (when C++11 support was
finally completed in gcc).

So, update our requirements for gcc to at least 4.8.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-10-27 10:00:29 +01:00
Yann E. MORIN 9c0c7846cd support/dependencies: don't check for python on the host
We no longer have anything that needs it during the build, so we don't
require it anymore.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-10-26 22:52:15 +02:00
Yann E. MORIN 2496189a42 core: drop check-uniq-files
Back a few years ago, when we were starting to think about top-level
parallel build, we were not sure how to deal with packages that
installed the same files, so we wanted to catch the situation to assess
how prevalent that was, before we decided what to do and how to address
it.

However, the trend nowadays is that packages will install in a
per-package target/ (and staging/ and host/), and the final directories
will be assembled in a reproducible (alphabetical) order, so if two
packages install the same file, the last one will win (as is currently
the case).

Besides, check-uniq-files reports loads of spurious errors when packages
get reinstalled (e.g. during development).

Finally, check-uniq-files is the only script called during the build,
that is written in python.

So, get rid of check-uniq-files.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-10-26 21:19:07 +02:00
Peter Korsgaard 1cab59a5e8 package/docker-compose: bump version to 1.24.1
Fixes https://gitlab.com/buildroot.org/buildroot/-/jobs/332656041

The recent bump of a number of python packages broke docker-compose, as
docker-compose specifies both minimum and maximum versions for (most of) its
dependencies:

Dependencies of docker-compse 1.20.1 (! = unmet):
cached-property: < 2 (currently 1.51)
docopt: < 0.7 (currently 0.6.2)
! pyyaml: < 4.0, patched to < 4.3 (currently 5.1.2)
requests: < 2.19, patched to < 3 (currently 2.22.0)
! texttable: < 0.10 (currently 1.6.2)
websocket-client: < 1.0 (currently 0.56.0)
! docker: < 4.0 (currently 4.1.0)
dockerpty: < 0.5 (currently 0.4.1)
six: < 2 (currently 1.12.0)
jsonschema: < 3 (currently 2.5.1)
enum34: < 2 (currently 1.1.6)
backports.ssl-match-hostname: >= 3.5 (currently 3.7.0.1)
ipaddress: >= 1.0.16 (currently 1.0.23)

To fix this, bump docker-compose to the most recent release (1.24.1).  This
is unfortunately not enough, as our docker, pyyaml, requests and texttable
packages are too new, so add 3 patches from upstream to relax the version
checks of dependencies.  Notice that patch 0003 is from
https://github.com/docker/compose/pull/6623 and has not been merged yet.

Discussions around the problem of these maximum versions of the dependencies
and the fact that all downstream users have to patch it is ongoing here:

https://github.com/docker/compose/issues/6756

docker-compose 1.24.1 added a requirement for ssh support in python-docker in:
7b82b2e8c7

So add a dependency for python-paramiko and update the toolchain dependency
for C++ (from python-paramiko -> python-cryptography) and adjust the
toolchain configuration of the runtime test to match.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-26 20:17:14 +02:00
Francois Perrad 76b4fcb5dc support/testing: add luvi test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-26 16:33:12 +02:00
Francois Perrad d1e01107a7 support/testing: add lua-sdl2 test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-26 16:31:36 +02:00
Francois Perrad 308ef3166b support/testing: add lua-gd test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-26 16:31:03 +02:00
Francois Perrad eaa2c3a6f7 support/testing: add lzlib test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-26 16:30:58 +02:00
Ricardo Martincoski f3c4a9e1cd support/testing: test_syslog_ng: improve commands
There is no need for double grep, so choose a better regexp. Use &&
instead of ; between commands so the sequence of commands fail faster.

Break the last sequence of commands in 2 calls run() so the proper
return code can be tested for each.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-16 21:33:04 +02:00
Ricardo Martincoski d3be2087d0 support/testing: add tmux test
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-12 15:37:51 +02:00
Francois Perrad 602f0061ff support/scripts/graph-depends: cut on host-ccache
When selected, host-ccache is a dependency of almost all packages.
As such, it clutters the dependency graph uselessly.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Reviewed-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-02 21:07:14 +02:00
Marcin Niestroj f8b8a9c529 support/testing: add turbolua test
Signed-off-by: Marcin Niestroj <m.niestroj@grinn-global.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-10-02 20:57:08 +02:00
Francois Perrad 101ae29c07 support/testing: add lua-cqueues test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-29 09:15:08 +02:00
Carlos Santos 9e546440d3 support/scripts/check-kernel-headers.sh: use a trap to remove the temporary file
The POSIX specification defines a 'trap <action> EXIT' mechanism that is
useful to perform clean-up actions in shell scripts. A trap has two main
advantages over hand-crafted clean-up mechanisms:

- It runs even if the process is terminated by a SIGTERM.
- It runs even if the script stops due to a pipeline failure (set -e).

Now we can make the script to stop immediately if a compilation error
occurs, instead of letting it try to run an unexisting program.

This change may appear to be overkill but Buildroot is an open source
project and each piece of code is a potential learning tool for other
developments. We must strive to provide good examples.

Signed-off-by: Carlos Santos <unixmania@gmail.com>
Acked-by: Yann E. MORIN <yann.morin@orange.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-25 22:07:29 +02:00
Carlos Santos 6136765b23 toolchain: generate check-headers program under $(BUILD_DIR)
Some installations mount /tmp with the 'noexec' option, which prevents
running the program generated there to check the kernel headers.

Avoid the problem by generating the program under $(BUILD_DIR), passed
as the first argument to check-kernel-headers.sh.

We could globally export a TMPDIR environment variable with some path
under $(BUILD_DIR) but such solution would be too intrusive, depriving
the user from the freedom to set TMPDIR at his will (or needs).

Fixes: https://bugs.busybox.net/show_bug.cgi?id=12241

Signed-off-by: Carlos Santos <unixmania@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-25 22:07:24 +02:00
Thomas Petazzoni f89871e810 support/config-fragments: fix br-riscv{32,64} toolchain fragments
Since commit aee39cbf27 ("arch/riscv:
set the default float ABI based on ISA extensions"), RISC-V 32/64 use
the lp32d/lp64d ABIs by default. But our pre-built external toolchains
were built with the LP32/LP64 ABI.

Building with lp32d/lp64d gcc flags, but a toolchain built with the
LP32/LP64 ABI causes a number of failures such as:

  /home/mark/buildroot-test/instance-1/output/host/riscv64-buildroot-linux-gnu/sysroot/usr/include/gnu/stubs.h:11:11: fatal error: gnu/stubs-lp64d.h: No such file or directory

or:

  /home/mark/buildroot-test/instance-1/output/host/opt/ext-toolchain/bin/../lib/gcc/riscv64-buildroot-linux-gnu/7.4.0/../../../../riscv64-buildroot-linux-gnu/bin/ld: /tmp/cc2BTtFE.o: can't link hard-float modules with soft-float modules
  /home/mark/buildroot-test/instance-1/output/host/opt/ext-toolchain/bin/../lib/gcc/riscv64-buildroot-linux-gnu/7.4.0/../../../../riscv64-buildroot-linux-gnu/bin/ld: failed to merge target specific data of file /tmp/cc2BTtFE.o

So let's fix our config fragments to reflect the ABIs those toolchains
were built with.

Fixes:

  http://autobuild.buildroot.net/results/a3959b0613cf561059483abc580b144be4817d1a/ (libsepol)
  http://autobuild.buildroot.net/results/3db50d8a0a913413b2198d6c301419136d2d22a7/ (attr)
  http://autobuild.buildroot.net/results/7780fada05b8440ae3e97618615624a6a2dac03f/ (libusb)
  and many others

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-23 21:56:24 +02:00
Sergey Matyukevich 2a220de466 binaries-marvell: bump version and switch to common firmware
Since release 18.12 binaries-marvell repository provides
common firmware supporting both A7K and A8K SoC families.

This commit bumps package version to 18.12 and removes
platform specific binary selections from Config.in.
Single firmware image suitable for both A7K and A8K
platforms is now specified in mk file explicitely.

Legacy handling is not needed, as configs which did have
the option set will continue to work without change.

Signed-off-by: Sergey Matyukevich <geomatsi@gmail.com>
[Arnout:
 - Remove BINARIES_MARVELL_IMAGE entirely;
 - Add remark about legacy handling;
 - Remove the deprecated option from the defconfigs and test that use
   it.]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-09-21 18:31:26 +02:00
Adrian Perez de Castro e0c879509d support/dependencies/dependencies.sh: check for JSON:PP Perl module
The JSON::PP Perl module is used at build time by the webkitgtk and
wpewebkit packages.

Signed-off-by: Adrian Perez de Castro <aperez@igalia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-17 22:36:42 +02:00
Thomas Petazzoni ffcd34af07 support/scripts/pkg-stats: simplify Git commit id retrieval
As suggested by Baruch Siach, using "git rev-parse HEAD" is a lot
simpler than playing around with "git log" to just retrieve the commit
id corresponding to the current HEAD.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-15 16:10:22 +02:00
Thomas Petazzoni 3f08ffa423 support/scripts/pkg-stats: extract current commit id, not master
pkg-stats extracts the Buildroot commit id from which the package
information was collected. However, when doing so, it always assumes
we're using the master branch, by running "git log master".

But in fact, pkg-stats can be run from any branch/tag, so it makes a
lot more sense to use "git log HEAD".

Cc: victor.huesca@bootlin.com
Cc: Baruch Siach <baruch@tkos.co.il>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-09-12 18:53:14 +02:00
Peter Korsgaard d8e6851f11 Merge branch 'next' 2019-09-03 15:03:02 +02:00
Peter Korsgaard 1fcdfbfb8a Update for 2019.08
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-09-01 23:06:01 +02:00
Yann E. MORIN 20cbf17e0a support/graph-size: reorder colours assigned to sizes
Now that we can order packages from biggest to smallest, it makes sense
to assign the most aggressive colours to the biggest packages.

As such, reorder the current colours so that we have, in order:
  - red-ish
  - orange-ish
  - yellow-ish
  - purple-ish
  - eggplant-ish (is that even a colour? :-] )
  - some-indeterminate-blue-ish
  - dark-green-ish
  - light-green-ish

For the previous, smallest-first ordering, it does not matter much what
the ordering is: the actual colours are still somewhat-unpredictably
assigned to packages, depending on the cut-off limit...

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:51:47 +02:00
Yann E. MORIN 33c1ef88f8 support/graph-size: add option to sort packages in reverse size order
Currently, the packages are sorted smallest first, and biggest last
(with unknown and others second-to-last and last, resp.).

Add an option to invert the ordering (but keeping unknown and others at
their current positions).

This has the nice side effect that we can now control the colours
assigned to the biggest package(s), as the colours are cycled from the
first to the last. Currently, the biggest packages gets a redish colour,
which is appropriate, but the second gets a greenish one, which is not
as appropriate (but changing that can come later).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:50:05 +02:00
Yann E. MORIN 1dbce133db support/graph-size: add option to report size with IEC prefixes
When dealing with embedded devices, storage is more often than not some
kind of flash device, on which the memory is usually counted as powers
of 1024 instead of powers of 1000. As such, people may prefer reports
using IEC prefixes [0] instead of the SI prefixes.

Add an option to that effect.

We use argparse's ability to use custom actions [1] [2], to provide a
set of options that act on a boolean, but has a single help entry and
internally ensures consistency of the settings. We could have been using
the more conventional store_true/store_false actions instead, but that
would have meant either two help entries, one for each set of options,
and/or some logic after parse_args() to check the validity of the
settings.

[0] https://en.wikipedia.org/wiki/Binary_prefix
[1] https://docs.python.org/2/library/argparse.html#action
[2] https://docs.python.org/2/library/argparse.html#argparse.Action

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:49:22 +02:00
Yann E. MORIN e9cdabee71 support/graph-size: add option to change percentage to group in Others
Currently, we group packages that contribute less then 1%, into the
"Other" category.

However, in some cases, there can be a lot of very comparatively small
packages, and they may not exceed this limit, and so only the "Others"
category would be displayed, which is not nice.

Conversely, if there are a lot of packages, most of which only so
slightly exceeding this limit, then we get all of them in the graph,
which is not nice either.

Add a way for the developers to pass a different cut-off limit. As for
the dependency graph which has BR2_GRAPH_DEPS_OPTS, add the environment
variable BR2_GRAPH_SIZE_OPTS to carry those extra option (in preparation
for more to come, later).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
[Arnout:
 - remove empty base class definition from Config;
 - use parser.error instead of ValueError for invalid argument.]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:44:27 +02:00
Yann E. MORIN 3fc3c4ac99 support/graph-size: display human-readable size
Currently, we forcibly report sizes in multiple of Kilobytes. In some
big configurations, the sizes of the system as a whole, as well as that
of individual packages, may exceed megabytes, and when some artistic
assets get used, even the gigabyte may get exceed.

These big sizes are not easy to read when expressed in kilobytes.

Additionally, some very small packages might have sizes below the
kilobyte (and when we can specify the cut-off grouping size, they may
get reported), and thus the size displayed for those would be 0 kB.

Add a helper function that can format a floating-point size into a
string with all the appropriate formatting:

  - there are at least 3 meaningfull digits visible, i.e. we display
    "3.14" or "10.4" instead of just "3" or "10", but for big number we
    don't care about too many precision either, so we report "100" or
    "1000", not "100.42" or "1000.27";

  - the proper SI prefix is appended, if needed.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:15:35 +02:00
Yann E. MORIN e8de561436 support/graph-size: report 'Unknown" after all packages, but before "Others"
Currently, the "unknown" category may be reported anywhere, so it does
not really stand out when there are a lot of packages in the graph.

Move it towards the end, but right before the "other" category, so that
it is a bit more visible. Like for Others, don't report it if its size
is zero.

Also, make it title case (i.e. "Unknown" instead of "unknown").

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:12:37 +02:00
Yann E. MORIN c68ee73924 support/graph-size: don't report "Others" if size is zero
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:08:18 +02:00
Yann E. MORIN a2d20ca613 support/graph-size: introduce main()
It is nicer overall to have a main() function, like all our other
scripts tend to have too.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:08:01 +02:00
Yann E. MORIN cecaf7001f support/graph-size: fix flake8 warnings
There are three E501 warnings returned by flake8, when run locally,
because we enforce a local 80-char limit, but that are not reported by
the gitlab-ci jobs because only a 132-char limit is required there.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-26 22:07:50 +02:00
Victor Huesca 152776b420 support/testing: new test for root password
Add support to test that the root passowrd is working as expected.
- Buildtime test: Check the hash present in the generated '/etc/shadow'.
- Runtime test: Build an armv7 image and try to login with a password.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-13 22:30:41 +02:00
Ricardo Martincoski 620c98a214 support/testing: factor out run_cmd_on_host
Currently many test cases call subprocess.check_output on their own.
Factor out that code to an infra method so the call get standardized.

This will be handful when switching the test infra to use Python 3.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-11 22:11:58 +02:00
Ricardo Martincoski 1b2928201f support/testing: do not print non-utf-8 chars on cbor test
The python-cbor sample script to be run on the target dumps a binary
variable to the target stdout. This is done for debug purposes-only, in
the case the test on test infra fails.
This non-utf-8 is currently silently ignored by the Emulator class /
pexpect.spawn from the test infra because the infra uses Python 2, that
in turn do not differentiate between string and byte data.

Make the code Python3-friendly (Python 3 in the host) by doing the right
thing and encoding the data before printing it.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-11 22:11:55 +02:00
Ricardo Martincoski 8de4291bd2 support/testing: check ubi image before booting
Do the same as other fs tests and minimally check the ubi image before
booting.
The call to 'file' was already there, but the output wasn't tested for
some unknown reason. Add the assert for the output of the command.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-11 22:11:53 +02:00
Ricardo Martincoski a565917046 support/testing: provide entropy to perl tests
Newer versions of perl-io-socket-ssl require entropy.
Switch to use armv5 builtin kernel that already provides entropy for all
perl tests.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/269139402

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Francois Perrad <francois.perrad@gadz.org>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-11 14:17:28 +02:00
Ricardo Martincoski eb6b0fd87a support/testing: provide entropy to lua tests
Newer versions of lua-http require entropy.
Switch to use armv5 builtin kernel that already provides entropy for all
lua tests.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/269139374
https://gitlab.com/buildroot.org/buildroot/-/jobs/269139376

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Francois Perrad <francois.perrad@gadz.org>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-11 14:17:28 +02:00
Ricardo Martincoski d23056ffe1 support/testing: make crossbar to use umsgpack
Since version 19.3.3, python-autobahn assumes that when CPython is used
msgpack will be used as well [1]. But it still allows the user to
override this behavior by setting an environment variable [2] to use
umsgpack.

Make the test to explicitly use umsgpack since it is part of the minimal
config (python-crossbar selects python-u-msgpack).

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/269139430

[1] ea019b8042
[2] fe70ceebe0

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-11 14:17:28 +02:00
Ricardo Martincoski ef5e87bb92 support/testing: make treq tests to ignore warning
Newer versions of python-treq display a warning at runtime when
service_identity is not installed:
"Without the service_identity module, Twisted can perform only
rudimentary TLS client hostname verification."

This warning message confuses the test that looks for another string in
stdout. Make the test ignore other messages while still expecting
"Connection refused".

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/269139449
https://gitlab.com/buildroot.org/buildroot/-/jobs/269139450

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-11 14:17:28 +02:00
Ricardo Martincoski 5c39c90ae1 support/testing: fix hardening tests
Since "2467822c85 package/checksec: bump to version 2.1.0" the hardening
tests fail because upstream slightly changed the way the script is
called.
According to README.md: "- All options now require `--$option=$value`
instead of `--$option $value`"

Instead of just replacing '--output json' with '--output=json' take into
account that upstream also changed the usage example to show --format
instead of --output. Both options do exactly the same, but following the
usage example seems to be more future-proof.

Upstream also improved the json output. Now when a file is passed as
parameter, the json has the file name as the main key, instead of the
string "file". Adjust the test cases accordingly.

Fixes:
tests.core.test_hardening.TestFortifyConserv
tests.core.test_hardening.TestFortifyNone
tests.core.test_hardening.TestRelro
tests.core.test_hardening.TestRelroPartial
tests.core.test_hardening.TestSspNone
tests.core.test_hardening.TestSspStrong

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Cc: Matt Weber <matthew.weber@rockwellcollins.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-05 22:15:18 +02:00
Thomas Petazzoni dc938c221d support/config-fragments/autobuild: set gcc version for RISC-V toolchains
Prior to b3ba26150d
("toolchain/toolchain-external/toolchain-external-custom: be more
flexible on gcc version"), the default gcc version selected by
Buildroot for custom external toolchain was affected by the
BR2_ARCH_NEEDS_GCC_AT_LEAST_xyz definitions.

Since BR2_riscv selects BR2_ARCH_NEEDS_GCC_AT_LEAST_7, gcc 7.x was the
default gcc version assumed to be used in a custom RISC-V external
toolchain, so our config snippets for RISC-V toolchains were correct.

With b3ba26150d applied, the default gcc
version assumed for custom external toolchains is the latest one
(currently gcc 9.x), while our RISC-V toolchains use gcc 7.x. So we
now need to explicitly give the gcc version used by our RISC-V
toolchains, otherwise the build fails with:

  Incorrect selection of gcc version: expected 9.x, got 7.4.0

Fixes:

  http://autobuild.buildroot.net/results/b872befe1adec2633b9cbcc49bc0eb7619f606c2/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-04 08:55:10 +02:00
Yann E. MORIN cfb929fbfa core: allow br2-external trees to provide opensl
Similar to toolchains and jpeg, we now offer a way for br2-external
trees to provide their openssl implementation, which gets included in
the openssl choice.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-04 00:13:37 +02:00
Yann E. MORIN 3b67e8e664 core: allow br2-external trees to provide libjpeg
Similar to toolchains, we now offer a way for br2-external trees to
provide their libjpeg implementation, which gets included in the jpeg
choice.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-04 00:13:37 +02:00
Yann E. MORIN fa037acee0 core: allow br2-external trees to provide pre-configured toolchains
Since we have a choice for the pre-configured pre-built toolchains,
there is no possbility for a br2-external to provide its own. The
only solution so far for defconfigs in br2-external trees is to use
BR2_TOOLCHAIN_EXTERNAL_CUSTOM and define all the bits by itself...

This is not so convemient, so offer a way for br2-external trees to
provide such pre-configured toolchains.

To allow for this, we now scan each br2-external tree and look for a
specific file, provides.toolchains.in. We generate a kconfig file that
sources each such file, and that generated file is sourced from within
the toolchain choice, thus making the toolchains from a br2-external
tree possible and available in the same location as the ones known to
Buildroot:

    Toolchain  --->
        Toolchain type (External toolchain)  --->
        Toolchain  --->
            (X) Arm ARM 2019.03
            ( ) Linaro ARM 2018.05
            ( ) Custom toolchain
                *** Toolchains from my-br2-ext-tree: ***
            ( ) My custom ARM toolchain
                *** Toolchains from another-br2-ext-tree: ***
            ( ) Another custom ARM toolchain
            ( ) A third custom ARM toolchain

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-04 00:13:37 +02:00
Yann E. MORIN edf32b021c core: split generated kconfig file
Currently, the kconfig part contains two things: the kconfig option
with the paths to br2-external trees, and the kconfig menus for the
br2-external trees.

When we want to include more kconfig files from the br2-external tree
(e.g. to get definitions for pre-built toolchains), we will need to
have the paths defined earlier, so they can be used from the br2-external
tree to include files earlier than the existing menus.

Split the generated kconfig file in two: one to define the paths, which
gets included early in our main Config.in, and one to actually define
the existing menus, which still gets included at the same place they
currently are.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-04 00:13:37 +02:00
Yann E. MORIN 0797dae894 core: prepare for generating multiple kconfig fragments
We currently redirect the output of each helper function. This was nice
as long as we were generating single .mk and .in fragments.

But we are soon to need more .in fragments.

So, do the redirection inside the .in helpers.

We do not (currently) need to generate more than one .mk fragment, but
for consistency, do the redirection in the .mk helper too.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 21:51:40 +02:00
Yann E. MORIN d027cd75d0 core: generate all br2-external files in one go
When we introduced support for multiple br2-external trees, we
introduced two files, one on the Makefile side, needed very early,
and one on the kconfig side, needed later in the configuration
process. We naturally introduced a two-step generation, as it looked
like the simplest and most obvious way.

But now, we are on the verge of generating more files on the kconfig
side, and it does not make sense to add even more steps to generate
them.

And even better yet, we can generate both the Makefile-side and
kconfig-side files at the same time, in fact.

Make it so.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 21:51:40 +02:00
Yann E. MORIN 2130903347 support/scripts/br2-external: drop help for internal helper script
We do not usually provide help for our internal scripts. Besides, such
help has a tendency to bitrot pretty quickly anyway.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 19:58:46 +02:00
Yann E. MORIN 3617f1350a support/scripts/br2-external: declare missing local variables
Commit b14b02698 (core/br2-external: restore compatibility with old
distros) switched to using 'eval' to emulate associative arrays, for
those distros too old to have bash-4+.

In so doing, it forgot to declare the new local variables in the
respective helper functions.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 19:58:17 +02:00
Frank Vanbever 07f31ee263 support/cmake: Explicitly set CMAKE_SYSTEM
Some packages test for CMAKE_SYSTEM explicitly[1]

CMAKE_SYSTEM is comprised of CMAKE_SYSTEM_NAME and CMAKE_SYSTEM_VERSION.
It defaults to CMAKE_SYSTEM_NAME if CMAKE_SYSTEM_VERSION is not set[2]

At the point CMAKE_SYSTEM_NAME is set to "Linux" CMAKE_SYSTEM is already
constructed. Setting it explicitly ensures that it is the correct value.

This is because we do set CMAKE_SYSTEM_NAME twice, in fact:

  - first in toolchainfile.cmake, so that we tell cmake to use the
    "Buildroot" platform,

  - second, in the Buildroot.cmake platform definition itself, so that
    we eventually behave like the Linux platform.

We also set CMAKE_SYSTEM_VERSION to 1, and so the real CMAKE_SYSTEM
value should be set to Linux-1 if we were to follow the documentation to
the letter.

However, for Linux, the version does not matter, and in some situations
may even be harmful (that was reported in one of the commits that
introduce Buildroot.cmake and toolchainfile.cmake).

[1] Fluidsynth 0cd44d00e1/CMakeLists.txt (L80)
[2] https://cmake.org/cmake/help/git-master/variable/CMAKE_SYSTEM.html#variable:CMAKE_SYSTEM

Signed-off-by: Frank Vanbever <frank.vanbever@mind.be>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
[Peter: update commit message with description from Yann]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 19:17:32 +02:00
Ricardo Martincoski 1ad5e0a6a6 support/testing: test check-package ignores external.mk
Check external.mk is ignored only when in the root path of a
br2-external.

Add a file called external.mk as a fixture to be used by the test case.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Arnout: wrap at 80 columns]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-03 17:17:29 +02:00
Ricardo Martincoski 4323fbaf8d support/testing: add test for check-package
Check the basic usage for check-package.

It can be called using either absolute path, relative path or from PATH.
Files to be checked can be passed with either absolute path or relative
path (also including files in the current directory).

Also check it ignores some special files when checking intree files,
i.e. package/pkg-generic.mk, while still generating warnings for out-of-tree
files when called with -b.
In order to allow the later, add an empty line to the Config.in in the
br2-external being tested so the script does generate a warning.

Catches bug #11271.

More tests can be added later, for example compatibility to Python 3.

Suggested-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Arnout: wrap at 80 columns where appropriate; merge into a single
class.]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-03 17:17:29 +02:00
Ricardo Martincoski 8669243da3 support/testing: allow run-tests to be called from anywhere
Currently run-tests must be called from the Buildroot top directory.

Derive the top directory from the script path, so run-tests can be called from
any path.
As a consequence the test infra will always test the repo it belongs to.

Suggested-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-03 17:17:29 +02:00
Ricardo Martincoski 9609a7c2c2 support/testing: test atop with builtin kernel
Use the new builtin armv5 4.19 kernel to test atop.
The atop package cannot be tested using BASIC_TOOLCHAIN_CONFIG because
it needs kernel headers >= 3.14. So use an updated version of it,
copying the config fragment from
support/config-fragments/autobuild/br-arm-full.config

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 14:53:05 +02:00
Ricardo Martincoski 650e9903a9 support/testing: drop haveged for python tests
The new builtin armv5 kernel 4.19 uses VirtIORNG to provide entropy, so we
no longer need haveged.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/264508536
https://gitlab.com/buildroot.org/buildroot/-/jobs/264508537
https://gitlab.com/buildroot.org/buildroot/-/jobs/264508542
https://gitlab.com/buildroot.org/buildroot/-/jobs/264508543

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[Peter: reword]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 14:53:05 +02:00
Ricardo Martincoski 56e38a705e support/testing: test syslog-ng with builtin kernel
Use the new builtin kernel 4.19 with VirtIORNG to provide entropy to
test syslog-ng.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 14:53:05 +02:00
Ricardo Martincoski 7acb32dabb support/testing: add builtin armv5 kernel 4.19 with entropy
More and more packages being tested by the test infra, e.g. syslog-ng,
need entropy at startup, usually reading from /dev/random.

Some test cases can also depend on a kernel version newer than the
builtin ones already provided by the test infra:
 - 3.11.0 for armv5;
 - 4.0.0 for armv7.

Add a new builtin kernel to be used by such test cases.
Add it for armv5 so most test cases that switch to use this kernel can
keep using BASIC_TOOLCHAIN_CONFIG.
Use the same kernel version and kernel config as qemu_arm_versatile plus
HW_RANDOM_VIRTIO for VirtIORNG to be usable.
Copy the actual binary file from the syslog-ng runtime test at current
master @ 29e1cb8884.

Since there is already a 'kernel-versatile' file on autobuild.buildroot.net
and we must keep it with this name for reproducibility purposes, create a
simple naming convention for newer builtin kernel images and dtb files:
kernel-<defconfig>-<kernel_series_version>
<dtb_name>-<kernel_series_version>.dtb
Pass '-device virtio-rng-pci' to qemu when this kernel is used.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[Peter: use this new kernel instead of the old builtin/armv5 kernel]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-03 14:53:05 +02:00
Thomas Petazzoni 228b0b1f00 support/config-fragments/autobuild: update CSV after musl toolchain rename
In commit a589325405
("support/config-fragments/autobuild: rename br-riscv64-musl config"),
the RISC-V 64-bit musl toolchain config snippet was renamed, but the
toolchain.csv file was not updated accordingly.

Due to this, utils/genrandconfig was no longer able to generate any
configuration.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-03 12:57:46 +02:00
Thomas Petazzoni b96c343b2d support/config-fragments/autobuild: make br-arm-internal-glibc bleeding edge
We do not have any configuration that tests the very bleeding edge gcc
and binutils versions, so let's change br-arm-internal-glibc to use
the latest version of gcc (9.x right now) and binutils (2.32 right
now). The idea is that this defconfig should be updated to the latest
version of gcc and binutils when their version is bumped.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-02 23:33:00 +02:00
Thomas Petazzoni 84f356e375 support/config-fragments/autobuild: update toolchains to 2019.05.1
All toolchains have been rebuilt with Buildroot 2019.05.1. A number of
toolchains are now using Linux headers 5.1 instead of 4.19, because
5.1 is now the default version.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-02 23:32:40 +02:00
Thomas Petazzoni a589325405 support/config-fragments/autobuild: rename br-riscv64-musl config
For internal toolchains, we have a policy of naming the files with
"internal", to clearly distinguish them from external toolchain
configurations.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-02 23:32:19 +02:00
Francois Perrad 1a49dcb528 support/testing: improve test_luasyslog
Lua has a builtin lsyslog module, so let's test this one as well.

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-08-02 17:35:44 +02:00
Mark Corbin 1003af2ebe support/config-fragments: add RISC-V 64-bit musl autobuild config
Add a RISC-V 64-bit autobuild configuration for the internal
toolchain with musl.

Signed-off-by: Mark Corbin <mark.corbin@embecosm.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-01 23:55:40 +02:00
Ricardo Martincoski 450863097f support/testing: use virtio-rng-pci to test syslog-ng
Recent versions of syslog-ng need some entropy on startup.
So use VirtIORNG to provide it. In order to accomplish this:
 - build the kernel containing the driver;
 - pass '-device virtio-rng-pci' to qemu.

Use the same kernel version and kernel config as qemu_arm_versatile.
It already has PCI enabled but it does not have HW_RANDOM_VIRTIO, so add
a defconfig fragment to enable the drivers.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/259856394

At the same time, fix a typo (missing '#') that resulted in the
generation of root.tar. This file is not used in the test.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-01 20:31:54 +02:00
Francois Perrad 0b32cd888e package/luaposix: bump to version 34.1.1
the module std.normalize is no longer a dependency

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 18:11:33 +02:00
Victor Huesca 294fc3218c support/scripts/pkg-stats: retrieve packages latest version using processes
The major bottleneck in pkg-stats is the time spent waiting for
answers from remote servers. Two functions involve such communication
with remote servers:

- 'check_package_urls' which checks that each package upstream website
  is up, it is efficient due to the use of process-pools thanks to
  Matt Weber.

- 'check_package_latest_version' which fetches the latest package
  version from release-monitoring, it uses a http-pool but runs
  sequentially.

This patch extends the use of process-pools to 'check_latest_version'.
Due to some limitations of multiprocess callbacks, this patch loses
the overall progress of packages in favour of just the current package
name.

Runtimes for this function are ~3m vs ~25m for the linear version.
Tested on an i7 7500U (2/4 cores/threads @3.5GHz) with 15ms ping.

Note: There have already been work trying to parallelize this function
using threads but there were a failure on some configurations [1].
This implementation rely on a dedicated module already in use on this
script, so it's unlikely to see failure with this version.

[1] http://lists.busybox.net/pipermail/buildroot/2018-March/215368.html

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 18:04:09 +02:00
Victor Huesca 3938afe1b5 support/scripts/pkg-stats: fix flake8 issues
Fixes:
 - blank space before ':'
 - unused 'o' variable left from a previous patch
 - bad continuous alignment

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 18:04:09 +02:00
Victor Huesca 46190a36d9 support/scripts/pkg-stats: improve 'package_init_make_info'
The pkg-stats calls 3 times `make` to get a bunch of variables. These
variables can be obtained in only one make invocation.  This patch
replaces the three calls by just one and adjusts the parsing logic
accordingly.

Note: another option suggested by Arnout would be to run `make
show-info` that produces a json with the necessary variables.  This
would avoid the duplicated effort done in pkg-stats and pkg-utils and
allow to add other infos to pkg-stats like dependencies, reversed
dependencies or if the package is virtual.

In order to use this method, the following changes are required in
pkg-generic's show-info:

 - include license_files;
 - have an option to run it on *all* packages, not just the selected
   ones.

This patch take the simplest approach of only factorizing the make
calls as it requires less changes.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 14:27:23 +02:00
Thomas Petazzoni 2a16a0ff08 support/scripts/pkg-stats: rename dump_gen_info()
Since it's used only for the HTML output, and all other functions used
for HTML output are prefixed by dump_html, let's do so for
dump_gen_info() as well by renaming it to dump_html_gen_info().

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 11:10:41 +02:00
Victor Huesca 3c9d408207 support/scripts/pkg-stats: factorize date and commit
The 'dump_html' and 'dump_json' both include commit infos as well as the
current date. It make more sense to retrieve these information once.
This patch simply does this factorization.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 11:10:41 +02:00
Victor Huesca 500e1d6241 support/scripts/pkg-stats: add support for json output
Pkg-stats is a great script that get a lot of interesting info from
buildroot packages. Unfortunately it is currently designed to output a
static HTML page only. While this is great to include on the
buildroot's website, the HTML is not designed to be easily parsable and
thus it is difficult to reuse it in other scripts.

This patch provide a new option to output a JSON file in addition to the
HTML one.

The old 'output' option has been renamed to 'html' to distinguish from
the new 'json' option.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 11:10:41 +02:00
Victor Huesca 365aee0f38 support/scripts/pkg-stats: improve argparse usage
Move the mutual exculsion of the '-n' and '-p' options to be part of the
parser instead of being checked in main.

Signed-off-by: Victor Huesca <victor.huesca@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-08-01 11:10:41 +02:00
Francois Perrad a0b0976c93 support/testing: add luaposix test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-08-01 10:11:18 +02:00
Daniel J. Leach 6aee78c894 support/testing: openjdk JNI test cases
This test case builds a native library and ensures a Java class can load
and interact with the native library. The test also verifies Java code
can make system calls via the native library.

Signed-off-by: Daniel J. Leach <dleach@belcan.com>
Acked-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Tested-by: Adam Duskett <aduskett@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-06-30 17:32:34 +02:00
Thomas Petazzoni 345c29a4b7 support/testing/infra/emulator: increase memory size used by Qemu
By default, Qemu emulates a system with 128 MB of RAM. This is not
sufficient for some test cases we have, such as TestPerlDBDmysql,
where the initramfs is quite large. Therefore, this commit extends the
RAM size emulated by Qemu to 256 MB.

Fixes:

  https://gitlab.com/buildroot.org/buildroot/-/jobs/237108668

Thanks to Arnout for the analysis of the issue.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-06-22 19:28:36 +02:00
Francois Perrad c8ba0f7488 package/luasyslog: new package
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-06-20 18:25:15 +02:00
John Keeping 8dd1a41630 support/download/git: fix formatting of error message
'.' should be at the end of the sentence, not the beginning of a new
line.

Signed-off-by: John Keeping <john@metanate.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-06-20 18:14:49 +02:00
Peter Korsgaard f590097045 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-06-02 22:11:20 +02:00
Peter Korsgaard 8d4e26da08 Update for 2019.05
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-06-02 10:01:07 +02:00
Thomas Petazzoni ff9f778c66 support/gnuconfig: update to 2019-05-28
This update includes support for the C-SKY architecture.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
2019-05-31 22:59:52 +02:00
Guo Ren 2213636675 support/gnuconfig/config.sub: add C-SKY support
Modify config.sub so that it knows about the C-SKY
architecture. Without this, all autotools projects fail to build on
C-SKY.

Signed-off-by: Guo Ren <ren_guo@c-sky.com>
[Thomas: improved commit log]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-05-31 22:59:52 +02:00
Thomas Petazzoni 93e4f2d5ed support/gnuconfig: update README.buildroot with reality
4 out of 5 packages who are not using autotools but needed their
gnuconfig files updated were not complying with the recommandation in
support/gnuconfig/README.buildroot. The fifth package was converted to
be like the others: use UPDATE_CONFIG_HOOK as a <pkg>_POST_PATCH_HOOKS
rather than calling the CONFIG_UPDATE macro directly.

Now that all packages are consistent, update the README.buildroot file
to match the reality.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-05-30 12:33:34 +02:00
Guo Ren 910b431d63 support/gnuconfig/config.sub: add C-SKY support
Modify config.sub so that it knows about the C-SKY
architecture. Without this, all autotools projects fail to build on
C-SKY.

Signed-off-by: Guo Ren <ren_guo@c-sky.com>
[Thomas: improved commit log]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-05-27 22:29:07 +02:00
Alex Xu b7fddc0622 support/scripts/check-bin-arch: ignore /usr/lib/grub
/lib/grub is already ignored, so add /usr/lib/grub to support
BR2_ROOTFS_MERGED_USR.

Signed-off-by: Alex Xu <alex_y_xu@yahoo.ca>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-05-20 22:33:14 +02:00
Thomas Petazzoni 17809276b5 support/testing/tests: drop Python 2.x test in test_ipython
Since commit 6ebaef3818
("package/python-ipython: bump to version 7.4.0"), ipython is no
longer available for Python 2.x, as it requires Python 3.x.

However, the corresponding test case that was testing iPython under
Python 2.x was not removed at the same time, causing a failure of
TestIPythonPy2 test. Let's drop the test that is no longer relevant.

Fixes:

  https://gitlab.com/buildroot.org/buildroot/-/jobs/210208754

Cc: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Cc: Andrey Smirnov <andrew.smirnov@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-05-18 22:41:22 +02:00
Vadim Kochan 28878798cf support/dependencies: Add hint for ArchLinux when host needs IA32 libs
Add hint about which package needs to be installed to provide IA32 libs
support for the host when it is needed.

Signed-off-by: Vadim Kochan <vadim4j@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-05-10 11:04:09 +02:00
Yann E. MORIN d901aa32d5 support/scripts: use show-info to extract dependency graph
Currently, we extract the dependency graph from the aptly named but
ad-hoc show-dependency-graph rule.

We now have a better solution to report package information, with
show-info.

Since show-dependency-graph never went into a release so far, and
show-info does provide the same (and more), switch to using show-info.

Thanks to Adam for suggesting the coding style to have a readable code
that is not ugly but still pleases flake8. Thanks to Arnout for
suggesting the use of dict.get() to further simplify the code.

Note: we do not use the reverse_dependencies field because it only
contains those packages that have a kconfig option, so we'd miss most
host packages.

Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Adam Duskett <aduskett@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-05-07 23:03:41 +02:00
Nylon Chen 31161bfe9c support/config-fragments/autobuild: test the Andes nds32 architecture
Signed-off-by: Che-Wei Chuang <cnoize@andestech.com>
Signed-off-by: Greentime Hu <greentime@andestech.com>
Signed-off-by: Nylon Chen <nylon7@andestech.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-04-17 09:27:21 +02:00
Yann E. MORIN 9bde598ced infra/utils: add helper to generate comma-separated lists
Add a helper macro that, from a space-separated list of items, returns a
comma-separated list of the quoted items.

This will be useful when we need to generate lists in JSON, later...

Code suggested by Thomas P.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-15 23:24:13 +02:00
Yann E. MORIN 63cb953d14 support/scripts/brpkutil.py: wrap at 80 columns
Previously, the flake8 script didn't help us to detect when Python
scripts were incorrectly wrapped. Now, however, it does report such
errors.

Fix one such an error now.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Arnout: give commit message a more positive tone]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-13 15:59:47 +02:00
Romain Naour ec68531774 support/testing: add test for xserver/Mesa OpenGL/glxinfo
This test allow to check if the xserver with GLX is working properly.
This is a basic test but it allow to trigger the current bug reported
by [1].

To test if the glxinfo test is working, you can change "-display :0" by
"-display :1" in the glxinfo command line.

[1] https://bugs.buildroot.org/show_bug.cgi?id=11591

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Youssef Harmouch <youssef.harmouch@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-04-11 18:23:40 +02:00
Markus Mayer 09cb4ea933 support/dependencies/dependencies.sh: simplify an error message
There is no need to break the "\n" sequence using "%sn". We can just
escape it. Note: the escaping backslash needs to be escaped too,
because the shell will process the string before printf gets to see it.

Signed-off-by: Markus Mayer <mmayer@broadcom.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-04-10 07:45:30 +02:00
Francois Perrad 47db24b93a support/testing/tests/package/test_perl_*: regeneration
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-31 14:01:50 +02:00
Francois Perrad 6aaba51f3b support/testing/tests/package/test_lua*: regeneration
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-31 12:15:02 +02:00
Francois Perrad 5593ed554f support/testing: add lua-http test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-28 19:01:00 +01:00
Francois Perrad 6de6f21dc3 support/testing: add rings test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-28 17:20:00 +01:00
Francois Perrad 7ad1a47f9f support/testing: add luasocket test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-28 17:20:00 +01:00
Francois Perrad 97032e2f4f support/testing: add luasec test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-28 17:20:00 +01:00
Francois Perrad d2db91b334 support/testing: add luaossl test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-28 17:20:00 +01:00
Francois Perrad 3a3a661303 support/testing: add luafilesystem test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-28 17:20:00 +01:00
Francois Perrad 596465e61a support/testing: add luaexpat test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-28 17:20:00 +01:00
Francois Perrad f4dc945fbc support/testing: add lua-utf8 test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-28 17:20:00 +01:00
Francois Perrad f830d89121 support/testing: add lua-curl test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-28 17:20:00 +01:00
Francois Perrad e7e6b2478d support/testing: add lsqlite3 test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-28 17:20:00 +01:00
Francois Perrad ac2bfe7643 support/testing: add lpeg test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-28 17:20:00 +01:00
Yann E. MORIN 9c32b9286d support/graph-depends: use the new make-based dependency tree
Now that we can get the whole dependency tree from make, use it to
speed up things considerably.

So far, we had three functions to get the dependencies information:
get_depends(), get_rdepends(), and, somehow unrelated, get_version().

Because of the way %-show-{,r}depends works, getting the dependency tree
was expensive, the three functions all took a set of packages for which
to get the dependencies, in an attempt to limit the time it took to get
that tree, but we still had to call these functions iteratively, until
they returned no new dependency. This was pretty costly.

Now, getting the tree is much, much less costly, and we can get the
whole tree as cheaply as we previously got only the first-level
dependencies.

Furthermore, we can now also get the version information at the same
time, and that also brings in whether the package is virtual or not,
target or host.

So, we drop all three helper functions, and replace them with a single
one that returns all that information in one go: full dependency trees
(direct and reverse), per-package type, and per-package version.

Note: since commit 2d29fd96a (pkg-virtual: remove VERSION/SOURCE),
virtual packages are no longer reported as having a 'virtual' version,
so have since been displayed as regular packages in the graphs. Although
noone complained, this patch incidentally restores the initial
behaviour, and virtual packages are now correctly displayed as such
again.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-25 19:31:02 +01:00
Yann E. MORIN 893dde0102 support/graph-depends: don't eliminate mandatory deps for reverse graphs
We we simplify the dependency graph, we try to remove so-called
mandatory dependencies from each package, and for each mandatory that
was thus removed, reattach it to the root-package of the graph.

This was made so that mandatory dependencies (which are dependencies of
all packages, or at least of a lot of packages) do not clutter the
dependency graph, but that they are still shown in the graph, as
dependencies of the root package.

However, these mandatory dependencies are only _direct_ dependencies.
As such, it does not make sense to reattach a mandatory dependency when
doing a reverse graph. Worse, it can actually be incorrect.

For example, 'skeleton' is a mandatory dependency, and as such is
removed from all packages. But when doing a reverse graph, skeleton is
now in the dependency chain of, e.g. skeleton-init-none; it should then
not be removed.

In short: the notion of mandatory dependencies does not make sense in
the case of a reverse graph.

Consequently, skip over the mandatory dependency removal when doing a
reverse graph.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-25 19:29:11 +01:00
Yann E. MORIN 132aa296f9 support/graph-depends: also cut on host-gzip
When host-gzip is needed, it is a mandatory dependency of all packages.
As such, drawing the dependency lines toward host-gzip would uselessly
clutter the graph.

So, like for the skeleton, host-skeleton, and host-tar, we cut the
dependency chains toward host-gzip.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-17 14:35:30 +01:00
Yann E. MORIN 3311aa8cf4 support/graph-depends: also cut on host-tar
When host-tar is needed, it is a mandatory dependency of all packages.
As such, drawing the dependency lines toward host-tar would uselessly
clutter the graph.

So, like for the skeleton and host-skeleton, we cut the dependency chains
toward host-tar.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-17 14:35:21 +01:00
Yann E. MORIN 946d34b52d support/graph-depends: also cut on host-skeleton
host-skeleton is a dependency of almost all packages, except a very few.
As such, it clutters the dependency graph uselessly.

Do with it as we do for the skeleton: cut the dependency chains.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-17 14:35:13 +01:00
Yann E. MORIN 66cf4201f9 support/graph-depends: add option to exclude mandatory deps
Some times, multiple dependency graphs for a set of packages (mostly
the application-level packages for the project) are included in reports
(e.g. delivery notes). Repeating the mandatory dependencies on all
those graphs is useless and clutters the important dependencies.

When we had only two such mandatory dependencies (toolchain, skeleton),
it was manageable to list them as manual exclusions:
    -x toolchain -x skeleton

But we now have quite a few such dependencies, and it becomes a bit more
cumbersome to manage, not counting the ones we may add in the future.

Add an option to exclude all those mandatory dependencies, to generate
neat graphs.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-17 14:35:03 +01:00
Thomas Petazzoni 1e414fbe9b support/graph-depends: make sure mandatory deps are displayed
The current graph-depends implementation filters out a number of
"mandatory" dependencies that all packages have: dependency on
"toolchain" and dependency on "skeleton".

Despite this filtering, in full graph dependencies, "toolchain" and
"skeleton" are still shown, because they are target packages, and
therefore appear in the result of "make show-targets". Thanks to this,
they will be visible as dependencies of the "ALL" node, which is the
root of the dependency tree.

However, as we are going to introduce host-skeleton as a "mandatory
dependency" to be filtered out, this is no longer going to work.

This commit adjusts the remove_extra_deps() function to ensure that
when a mandatory dependency is removed, this dependency exists between
the root of the dependency tree and the mandatory dependency.

This issue was noticed by Yann E. Morin, and this commit provides a
different implementation than what Yann proposed in
https://patchwork.ozlabs.org/patch/910453/.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
  - list mandatory deps before removing them
  - fix flake8 warnings
]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-17 14:34:02 +01:00
Adam Duskett df7bf8fb41 support/testing/tests/package/test_openjdk: new test
This test is a simple "Hello, World" integration test of the OpenJDK
package.

It compiles the Java app on the host, then runs it on an emulated
AARCH64 target and verifies "Hello, World" is printed.

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-17 14:02:18 +01:00
Romain Naour b60729784a support/dependencies: add a check for python3
Since version 2.29, glibc requires python 3.4 or later to build the
GNU C Library [1].

We add a new check to verify the version of python3 interpreter
installed on the host.  If no suitable python3 interpreter is found,
define BR2_PYTHON3_HOST_DEPENDENCY to add host-python3 in package
dependencies when needed.

[1] https://www.sourceware.org/ml/libc-alpha/2019-01/msg00723.html

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Tested-by: Adam Duskett <aduskett@gmail.com>
Reviewed-by: Adam Duskett <aduskett@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Thomas: drop not so useful comment in the .mk file, as suggested by
Yann E. Morin.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-03-15 22:26:40 +01:00
Peter Korsgaard b9674056fb Update for 2019.02
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-03-04 22:49:56 +01:00
Peter Korsgaard d987412147 config-fragments: drop old crosstool-ng toolchains
These toolchains are very old and cause a number of autobuilder failures
that doesn't happen with more recent toolchains:

Fixes (glibc 2.18 does not provide O_TMPFILE):
http://autobuild.buildroot.net/results/c49e8361a1d4406eefd8fc1b35c8e5b061aa403b

Fixes (x86 toolchain built without libquadmath):
http://autobuild.buildroot.net/results/2d9724f169ccd60c7feb1cb549f1e2e1e9219ac3/

Use Codesourcery ARM toolchain (GCC 4.8.3) to provide a test with a old GCC
version.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-03-04 16:26:20 +01:00
Ricardo Martincoski 7b2dd942d0 Revert "testing/tests/download: test git special ref"
This reverts commit 81771cfcdc.

The download of sha1 of a special ref currently works or not depending
on the git client version in use.
With git version 2.11.0 (present in the docker image) it does not work.
With git version 2.17.1 it works.

For the sake of reproducibility, remove this part of the TestGitRefs
test case until some code gets added to the download infra to handle
sha1 of a special ref for any git client version.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/158295269

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-03-01 11:09:41 +01:00
Fabrice Fontaine 2eb31511f5 support/dependencies: require at least CMake 3.8
gerbera package in version 1.3 unfortunately now requires CMake >= 3.8
for C++17 macros:
b5fd39f30f

So we need to bump our requirement from 3.1 to 3.8. If the host doesn't
have a CMake >= 3.8, Buildroot will build its own host-cmake package.

Also drop patch that relax cmake requirement on json-for-modern-cpp

Fixes:
 - http://autobuild.buildroot.org/results/6405647b47b132ff5d0d211b92d407322d52d507

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-03-01 10:10:30 +01:00
Thomas Petazzoni 78596e3851 support/config-fragments/autobuild: use external toolchains in RISC-V configs
This commit replaces the two RISC-V configurations used for the
autobuilders to use pre-built external toolchains rather than internal
toolchains. This saves quite a bit of build time in the autobuilders,
and also allows people to reproduce build issues in a much more
efficient way, since rebuilding the toolchain is not needed.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Mark Corbin <mark.corbin@embecosm.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-14 21:53:19 +01:00
Thomas Petazzoni 71e449d3a5 support/config-fragments/autobuild: update all pre-built Buildroot toolchains
All toolchains have been rebuilt with Buildroot 2019.02-rc1.

Changes:

- Toolchains that were using no-longer maintained kernel headers
  versions have been changed to use a variety of newer kernel headers
  versions (4.4, 4.9 or 4.14).

- Since gcc 7.x is now the default in Buildroot, most toolchains that
  simply use the default gcc version use 7.x instead of 6.x.

- br-arm-cortex-a9-glibc uses gcc 8.x, binutils 2.31 and kernel
  headers 4.20

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-14 21:52:40 +01:00
Peter Korsgaard 11c55c94da support/testing: build a glibc toolchain for docker / docker-compose tests
runc (which is a reverse dependency of docker-engine) is about to gain a
!uclibc dependency, so move to a glibc toolchain instead.

There are currently no prebuilt x86_64 / core2 / glibc toolchains available,
so instead use the internal toolchain backend to build one.

While we are at it, drop the infra.basetest.BASIC_TOOLCHAIN_CONFIG
reference, as that ARM toolchain configuration doesn't make any sense for
this x86-64 based test.

add docker / docker-compose tests

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-12 20:03:46 +01:00
Arnout Vandecappelle (Essensium/Mind) 3f6587266e support/scripts/pkg-stats: fix flake8 errors
Fixes the following flake8 warnings:

support/scripts/pkg-stats:34:2: W605 invalid escape sequence '\$'
support/scripts/pkg-stats:34:4: W605 invalid escape sequence '\('
support/scripts/pkg-stats:34:11: W605 invalid escape sequence '\$'
support/scripts/pkg-stats:34:13: W605 invalid escape sequence '\('
support/scripts/pkg-stats:34:32: W605 invalid escape sequence '\)'
support/scripts/pkg-stats:34:34: W605 invalid escape sequence '\)'
support/scripts/pkg-stats:35:2: W605 invalid escape sequence '\s'
support/scripts/pkg-stats:35:14: W605 invalid escape sequence '\S'
support/scripts/pkg-stats:35:17: W605 invalid escape sequence '\s'
support/scripts/pkg-stats:42:1: E302 expected 2 blank lines, found 1
support/scripts/pkg-stats:587:133: E501 line too long (157 > 132 characters)

Note that the "invalid escape sequence" errors work because Python
leaves the \ in place if it doesn't recognise the escape sequence. But
it's better practice to use a raw string for regular expressions.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-07 22:09:55 +01:00
Carlos Santos bf2a308578 skeleton-custom: use a script to check merged usr structure
Introduce support/scripts/check-merged-usr.sh, a script that check if a
given path complies to the merged /usr requirements:

    /
    /bin -> usr/bin
    /lib -> usr/lib
    /sbin -> usr/sbin
    /usr/bin/
    /usr/lib/
    /usr/sbin/

Use this script in skeleton-custom.mk instead of a bunch of variables
filled by $(shell ...) macros. The same script will be used to check
rootfs overlays, in a forthcoming change.

Signed-off-by: Carlos Santos <casantos@datacom.ind.br>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-06 17:03:30 +01:00
Thomas Petazzoni 2c74d0aabb support/scripts/pkg-stats: add latest upstream version information
This commit adds fetching the latest upstream version of each package
from release-monitoring.org.

The fetching process first tries to use the package mappings of the
"Buildroot" distribution [1]. This mapping mechanism allows to tell
release-monitoring.org what is the name of a package in a given
distribution/build-system. For example, the package xutil_util-macros
in Buildroot is named xorg-util-macros on release-monitoring.org. This
mapping can be seen in the section "Mappings" of
https://release-monitoring.org/project/15037/.

If there is no mapping, then it does a regular search, and within the
search results, looks for a package whose name matches the Buildroot
name.

Even though fetching from release-monitoring.org is a bit slow, using
multiprocessing.Pool has proven to not be reliable, with some requests
ending up with an exception. So we keep a serialized approach, but
with a single HTTPSConnectionPool() for all queries. Long term, we
hope to be able to use a database dump of release-monitoring.org
instead.

From an output point of view, the latest version column:

 - Is green when the version in Buildroot matches the latest upstream
   version

 - Is orange when the latest upstream version is unknown because the
   package was not found on release-monitoring.org

 - Is red when the version in Buildroot doesn't match the latest
   upstream version. Note that we are not doing anything smart here:
   we are just testing if the strings are equal or not.

 - The cell contains the link to the project on release-monitoring.org
   if found.

 - The cell indicates if the match was done using a distro mapping, or
   through a regular search.

[1] https://release-monitoring.org/distro/Buildroot/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-02-06 15:57:09 +01:00
Ricardo Martincoski e2d1c38074 .gitlab-ci.yml: use "extends" keyword
Replace all YAML anchors with the new "extends" keyword because it is
more readable and more flexible (it works across configuration files
combined with the new "include" keyword).

Readability is more meaningful in .gitlab-ci.yml.in.
In the part of .gitlab-ci.yml that is auto-generated by 'make
.gitlab-ci.yml' keep the keyword in the same line of the job name.
So instead of this:
 zynqmp_zcu106_defconfig:
     extends: .defconfig
 tests.boot.test_atf.TestATFAllwinner:
     extends: .runtime_test
Use this:
 zynqmp_zcu106_defconfig: { extends: .defconfig }
 tests.boot.test_atf.TestATFAllwinner: { extends: .runtime_test }
Do this to to keep .gitlab-ci.yml easier to be post-processed by a
script.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-06 11:40:28 +01:00
Ricardo Martincoski 81771cfcdc testing/tests/download: test git special ref
Add a special ref to the static repo and check on the git refs test case
the download of a git package:
 - with the sha1 of a special ref as version;

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Arnout: change to use the sha1 of a special ref instead of the name]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-06 11:13:20 +01:00
Ricardo Martincoski ed3cda38cd testing/tests/download: test git tag
Add a tag to the static repo and check on the git refs test case the
download of a git package:
 - with the name of a tag as version;
 - with the sha1 of a tag itself as version;
 - with the partial sha1 of a tag itself as version;
 - with the sha1 of a commit pointed by a tag as version;
 - with the partial sha1 of a commit pointed by a tag as version;
 - with the sha1 of a commit reachable only by a tag as version;
 - with the partial sha1 of a commit reachable only by a tag as version.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-06 10:57:26 +01:00
Matt Weber c6386be05b testing/download: GIT use BRConfigTest
Enables the test to use the new non-emulator base class which takes
significantly less test time.

Cc: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-06 10:40:15 +01:00
Ricardo Martincoski 0461ef4dea testing/infra: Add BRConfigTest as superclass of BRTest
The git tests don't need to do a full build, they only need to do a
configure and download and/or legal-info. More tests of that type will
be added in the future. Therefore, we want to have a test base class
that doesn't automatically do a full build in the setUp().

Add this new class as a superclass of the existing BRTest class, so we
don't need to update existing tests. Only the code in run-tests that
iterates over all subclasses of BRTest has to be adapted to use
BRConfigTest instead.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-06 10:40:15 +01:00
Ricardo Martincoski 9cc7a12bde testing/tests/download: test git submodules
Add two submodules as static repos, add a branch to the main static repo
and check on the git refs test case the download of a git package:
 - repo with submodule but without support in the package;
 - repo with recursive submodules with support in the package.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Arnout: remove handling of inconsistent tarball hashes - that's an
 actual bug that should be fixed]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-06 09:56:58 +01:00
Peter Korsgaard a9219660d3 support/testing: add docker / docker-compose tests
Build for x86-64 as public containers in general are only available for
x86-64.  Docker needs a number of kernel options enabled, so use a custom
kernel config based on the qemu one.

Docker needs entropy at startup, so enable the virtio-rng-pci device to
expose entropy to the guest.  The default RAM amount (128M) is not enough to
run docker / docker-compose, so bump to 512MB.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Reviewed-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-05 23:05:00 +01:00
Thomas De Schampheleire aa62b36456 support/download/scp: fix download with scheme prefix 'scp://'
The scp download helper is broken when the server URL starts with 'scp://'.
Such prefix is used in two situations:
1. to let FOO_SITE point to an scp location without explicitly having to set
   'FOO_SITE_METHOD = scp'

2. when BR2_PRIMARY_SITE or BR2_BACKUP_SITE points to an scp location. In
   this case, there is no equivalent of 'SITE_METHOD'.

Strip out the scheme prefix, similarly to how the 'file' download helper
does it. That helper has the same cases as above.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-05 20:32:53 +01:00
Peter Korsgaard 092b2b5ada support/testing: test_dropbear.py: drop ssh port forward to the host
It is not used and may conflict with other services running on port 2222 on
the host machine.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-05 19:04:07 +01:00
Peter Korsgaard cc84a0dac6 support/testing: test_dropbear.py: add test for ssh connection to localhost
Use sshpass to pass the password on the command line.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-05 19:01:55 +01:00
Peter Korsgaard a34005699b support/testing: test_dropbear.py: use a constant for the password
The password is used in multiple places, so add a constant for it instead of
hardcoding it multiple times.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-05 19:01:42 +01:00
Ricardo Martincoski 5692e6afa1 testing/tests/download: test git branch
Add a branch to the static repo and check on the git refs test case the
download of a git package:
 - with a sha1 reachable by a branch name, but not pointed by it, as
   version. This is the most common use case for git refs in the tree;
 - with a partial sha1 of a commit reachable by a branch as version;
 - with a sha1 of the commit head of a branch as version;
 - with a partial sha1 of the commit head of a branch as version;

Enforce the download always occurs by removing the BR2_DL_DIR used for
the tarballs generated by the git download infra.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-05 10:23:03 +01:00
Thomas De Schampheleire 44084aa981 support/scripts/setlocalversion: ignore user settings for Mercurial
setlocalversion will use 'hg id' to determine whether or not the current
revision is tagged. If there is no tag, the Mercurial revision is printed,
otherwise nothing is printed.

The problem is that the user may have custom configuration settings (in
their ~/.hgrc file or similar) that changes the output of 'hg id' in a way
that the script does not expect. In such cases, the Mercurial revision may
not be printed or printed incorrectly.

It is good practice to ignore the user environment when calling Mercurial
commands from a well-defined script, by setting the environment variable
HGRCPATH to the empty string. See also 'hg help environment'.

In the particular case of Nokia, a custom extension adds dynamic tags in the
repository, i.e. tags that are stored in a file external to the repository
and only visible when the extension is active. These tags should not
influence the behavior of setlocalversion as they are not official Buildroot
tags, i.e. even if a revision is tagged, the Mercurial revision should still
be printed.

Note that this still does not solve the problem where an organization adds
_real_ tags in their Buildroot repository. For example, there might be a
moving tag 'last-validated' or tags indicating in which product release that
Buildroot revision was used. In these cases, setlocalversion will still not
behave as expected, i.e. show the Mercurial revision.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-02-04 21:25:33 +01:00
Thomas De Schampheleire 57e6dcf5fb support/scripts/setlocalversion: fix detection of hg revision when _not_ on branch 'default'
When Buildroot is stored in a Mercurial repository on a branch other than
'default' ('master' in git terms), setlocalversion (used to populate
/etc/os-release) will incorrectly think that this is a tagged version and
will NOT print out the revision hash.

This is due to the fact that the output of 'hg id' is assumed to be
    "<revision> <tags-if-any>"
but when on a branch it actually is:
    "<revision> (<branch>) <tags-if-any>"

To let setlocalversion receive the output it expects, explicitly ask 'hg id'
to retrieve only the revision hash and any tags, ommitting any branch
information.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-02-04 21:25:33 +01:00
Ricardo Martincoski 0a90740488 testing/tests/download: test case for git refs
All upcoming tests for git refs will rely on the return code of make to
determine whether a git ref can be downloaded or not and also to
determine whether the downloaded content is correct (all of this taking
advantage of the check-hash mechanism already in place for git
packages).
So to avoid false results i.e. in the case the check-hash mechanism
become broken in the master branch, add some sanity checks before the
actual test of download git refs.

Add the minimum test case for git refs containing only sanity checks.
Reuse the commit in the static repo.
Add a br2-external with two packages to check that:
 - trying to download an invalid sha1 generates an error;
 - downloading a valid sha1 that contains unexpected content generates
   an error.

In order to ease the maintenance and review, each upcoming patch adding
checks to this test case will add at same time the commits to the static
repo, the equivalent packages to the br2-external and code to the test
case.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-04 20:44:29 +01:00
Thomas Petazzoni 7dd2f4f3e1 support/testing/tests/download/gitremote: remove trailing whitespace
Fixes the following flake8 warnings:

support/testing/tests/download/gitremote.py:29:67: W291 trailing whitespace
support/testing/tests/download/gitremote.py:30:60: W291 trailing whitespace

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-02-04 17:10:35 +01:00
Ricardo Martincoski f284b5e7a1 testing/tests/download: add git hash test
Add one test case to ensure the hash is checked for git packages:
 - correct hash;
 - wrong hash;
 - no hash file.

Add required infra:
 - a GitRemote class, that can start a git server in the host machine to
   emulate a remote git server under the control of the test;
 - a new base class, called GitTestBase, that inherits from BRTest and
   must be subclassed by all git test cases.
   Its setUp() method takes care of configuring the build with a
   br2-external, avoiding to hit http://sources.buildroot.net by using
   an empty BR2_BACKUP_SITE. It also avoids downloading not
   pre-installed dependencies (i.e. lzip) every time by calling 'make
   dependencies' using the common dl directory, and it instantiates the
   GitRemote object.

Besides the Python scripts, add some fixtures used during the tests:
 - a br2-external (git-hash) with one package for each part of the test
   case;
 - a static git bare repo (repo.git) to be served using GitRemote class.

Neither the br2-external nor the check hash functionalities are the
subject of these tests per se, so for simplicity limit the check to the
error codes and don't look for the messages in the log.

Thanks to Arnout for the hint about how to add a bare repo to test.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
[Arnout: split long line; reorder imports to satisfy flake8]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-04 16:52:58 +01:00
Bryce Ferguson 3f1b965bc3 infra: add force build flag for host dependencies
This commit adds a config option which will force buildroot to
build all host dependencies even if they are already present on the
host system. This may be a desirable option if different hosts are
used to build the same source. In this case, some packages will be
built on one host that are not built on another. This is problematic
if build source archives are cached afterwards for offline builds.

Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Bryce Ferguson <bryce.ferguson@rockwellcollins.com>
Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
[Peter: reword, drop exit 1, reshuffle]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-04 15:52:44 +01:00
Thomas De Schampheleire 8e3c632f7d support/scripts/graph-build-time: replace confusing colors
The color for 'extract' is very similar to the one for 'install-images'.
Both are cyan-like.

Replace the former by a pale blue to make all colors sufficiently distinct.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-04 15:23:27 +01:00
Mathias De Mare ec757a813b support/scripts/graph-build-time: add download times
Total build time also involves download. Getting a visibility on the impact
of that step can be important for users/admins, e.g. to evaluate different
methods of BR2_PRIMARY_SITE.

Colors used are some kind of purple (primary scheme) and light orange
(alternate scheme).

Signed-off-by: Mathias De Maré <mathias.de_mare@nokia.com>
[ThomasDS: rebase and update colors to avoid confusion]
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-02-04 15:22:59 +01:00
Adam Duskett 6809f38351 dependencies.sh: Check for a host python version >= 2.7
Older distributions such as CentOS6 come with python2.6, which causes build
failures in packages such as host-libglib2 because they require python2.7 and
above.

host-libglib2 will produce the error message:
/bin/sh: python2.7: command not found

Python2.7 is a hard-coded value in configure.ac. If one changes the value to
just "python," the following stack trace is produced:

Traceback (most recent call last):
  File "./gdbus-2.0/codegen/gdbus-codegen.in", line 55, in <module>
    self.outfile.write(LICENSE_STR.format(config.VERSION))
ValueError    : sys.exit(codegen_main.codegen_main())
zero length field name in format

Instead of supporting an ancient version of Python that had its support ended
in October os 2013, it would be more pragmatic only to support Python2.7 and
above.

Luckily; CentOS6 has the centos-release-scl repository, which allows users to
install python2.7, and Debian 8 comes with Python2.7 already, making this patch
relatively low impact.

Signed-off-by: Adam Duskett <Aduskett@gmail.com>
[Peter: only look at major.minor to handle x.y.z with z < 10]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-02-04 13:22:55 +01:00
Ricardo Martincoski 4d5d0124c7 support/testing/infra/basetest: support br2-external
Some upcoming test cases can use one or more br2-external trees as
fixtures that provide packages used only in runtime tests.

Add support for br2-external into the BRTest class. Any test case can
then provide a list of paths for being used as br2-external trees
during the build of the image to test.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Daniel J. Leach <dleach@belcan.com>
Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Tested-by: Matthew Weber <matthew.weber@rockwellcollins.com>
[Thomas: use named argument for make_extra_opts.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-01-29 22:16:33 +01:00
Ricardo Martincoski 07bbf1b4dd support/testing/infra/builder: configure and build with make target and environment
Make the builder able to call 'VAR1=1 make VAR2=2 target'.

Allow sending extra parameters to be added to the end of make command
line. Uses for these purposes:
 - to configure a br2-external, using the 'BR2_EXTERNAL="dir" variable.
 - to specify a make target, such as 'foo-source.'

Allow adding variables to the environment when calling make.
These added variables allow a user to override default values from BuildRoot,
such as 'BR2_DL_DIR="dl"'.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Daniel J. Leach <dleach@belcan.com>
Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-01-29 22:14:55 +01:00
Ricardo Martincoski e2b98dfede support/testing: add atop test
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-01-28 10:00:45 +01:00
Thomas Petazzoni b1e294cc15 support/scripts/check-host-rpath: document existing functions
As suggested by Arnout Vandecappelle, let's document the
elf_needs_rpath() and check_elf_has_rpath() functions, before we make
them a bit more complicated with per-package directory support.

Suggested-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-01-17 22:38:34 +01:00
Yann E. MORIN fc8a5f56b9 infra/pkg-cmake: use an obviously-invalid value for CMAKE_SYSTEM_VERSION
In 36568732e4, we expanded toolchain.cmake to also define the value for
CMAKE_SYSTEM_VERSION, as the cmake documentation states that it must be
manually defined when doing cross-compilation [0]:

    When the CMAKE_SYSTEM_NAME variable is set explicitly to enable
    cross compiling then the value of CMAKE_SYSTEM_VERSION must also
    be set explicitly to specify the target system version.

However, the fix in 36568732e4 uses the version of the kernel headers,
assuming that would be the oldest kernel we could run on. Yet, this is
not the case, because glibc (for example) has fallbacks to support
running on kernels older than the headers it was built against.

The cmake official wiki [1] additionally states:

  * CMAKE_SYSTEM_VERSION : optional, version of your target system, not
    used very much.

Folllowed a little bit below, by:

  * CMAKE_TOOLCHAIN_FILE : absolute or relative path to a cmake script
    which sets up all the toolchain related variables mentioned above

    For instance for crosscompiling from Linux to Embedded Linux on PowerPC
    this file could look like this:

        # this one is important
        SET(CMAKE_SYSTEM_NAME Linux)
        #this one not so much
        SET(CMAKE_SYSTEM_VERSION 1)

    [...]

Furthermore, using the kernel headers version can be a bit misleading (as
it really looks like is is the correct version to use when it is not),
while it is obvious that 1 is not really the output of `uname -r` and
thus is definitely not misleading.

Finally, random searches [2] about CMAKE_SYSTEM_VERSION, mostly only
turns up issues related with Windows, Mac-OS, and to a lesser extent,
Android (where it is forcibly set to 1), with issues realted to running
under just Linux (as opposed to Adnroid) mostly non-existent.

Consequently, we revert to using the value that is suggested in the
cmake WiKi, i.e. 1, and which is basically what we also used as a
workaround in the azure-iot-sdk-c paclkage up until d300b1d3b1.

A case were we will need to have a real kernel version, is if we one day
have a cmake-based pacakge that builds and installs a kernel module [3],
because it will need the _running_ kernel version to install it in
/lib/modules/VERSION/, but in that case it will anyway most probably
not be the headers version.

[0] https://cmake.org/cmake/help/v3.8/variable/CMAKE_SYSTEM_VERSION.html
[1] https://gitlab.kitware.com/cmake/community/wikis/doc/cmake/CrossCompiling
[2] https://duckduckgo.com/?q=CMAKE_SYSTEM_VERSION
[3] https://stackoverflow.com/questions/38205745/cmake-system-version-not-updated-for-new-kernel

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Samuel Martin <s.martin49@gmail.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-01-16 23:19:09 +01:00
Arnout Vandecappelle (Essensium/Mind) 36568732e4 package/pkg-cmake: add CMAKE_SYSTEM_VERSION to toolchainfile.cmake
Quoting the CMake documentation:

  When the CMAKE_SYSTEM_NAME variable is set explicitly to enable cross
  compiling then the value of CMAKE_SYSTEM_VERSION must also be set
  explicitly to specify the target system version.

Thus, we should also set CMAKE_SYSTEM_VERSION in toolchainfile.cmake. It
is supposed to be set to the value of `uname -r` on the target. We don't
have that exact value available (unless we build the kernel), but the
value of BR2_TOOLCHAIN_HEADERS_AT_LEAST contains the (minimum) version
of the kernel it will run on, so it should be OK for all practical
purposes.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-01-12 16:06:51 +01:00
Mark Corbin 7a389a6ec8 support/config-fragments: add RISC-V 32-bit to autobuild configs
Add a minimal RISC-V 32-bit autobuild configuration for the
internal toolchain with glibc.

Signed-off-by: Mark Corbin <mark.corbin@embecosm.com>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-01-10 21:30:21 +01:00
Mark Corbin ae99fa6d64 arch: add support for RISC-V 32-bit (riscv32) architecture
This enables a riscv32 system to be built with a Buildroot generated
toolchain (gcc >= 7.x, binutils >= 2.30, glibc only).

This requires a custom version of glibc 2.26 from the riscv-glibc
repository. Note that there are no tags in this repository, so the
glibc version just consists of the 40 character commit id string.

Thanks to Fabrice Bellard for pointing me towards the 32-bit glibc
repository and for providing the necessary patch to get it to build.

Signed-off-by: Mark Corbin <mark.corbin@embecosm.com>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-01-06 14:09:31 +01:00
Thomas De Schampheleire 4d6e20ff46 support/download: fix scp downloads
scp download is broken, because scp is called without filename argument and
only the server is specified. The call is:
    scp <server> <outputfile>

but should be:
    scp <server>/<filename> <outputfile>

Instead of assuming '-u' lists a full URL including filename (which it is
not), align with the wget helper where -u is the server URL and -f gives the
filename.

With this commit, an scp download can work if FOO_SITE_METHOD is explicitly
set to 'scp' and the server does not have a scheme prefix 'scp://'.
The next commit will handle the case where a scheme prefix is present.

Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Thomas: s/URL/URI/, as noticed by Yann.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2019-01-03 22:05:43 +01:00
Ricardo Martincoski e7e30455ef Makefile: offload .gitlab-ci.yml generation
GitLab has severe limitations imposed to triggers.
Using a variable in a regexp is not allowed:
|    only:
|        - /-$CI_JOB_NAME$/
|        - /-\$CI_JOB_NAME$/
|        - /-%CI_JOB_NAME%$/
Using the key 'variables' always lead to an AND with 'refs', so:
|    only:
|        refs:
|            - branches
|            - tags
|        variables:
|            - $CI_JOB_NAME == $CI_COMMIT_REF_NAME
would make the push of a tag not to trigger all jobs anymore.
Inheritance is used only for the second level of keys, so:
|.runtime_test: &runtime_test
|    only:
|        - tags
|tests.package.test_python_txaio.TestPythonPy2Txaio:
|    <<: *runtime_test
|    only:
|        - /-TestPythonPy2Txaio$/
would override the entire key 'only', making the push of a tag not to
trigger all jobs anymore.

So, in order to have a trigger per job and still allow the push of a tag
to trigger all jobs (all this in a follow up patch), the regexp for each
job must be hardcoded in the .gitlab-ci.yml and also the inherited
values for key 'only' must be repeated for every job.
This is not a big issue, .gitlab-ci.yml is already automatically
generated from a template and there will be no need to hand-editing it
when jobs are added or removed.

Since the logic to generate the yaml file from the template will become
more complex, move the commands from the main Makefile to a script.

Using Python or other advanced scripting language for that script would
be the most versatile solution, but that would bring another dependency
on the host machine, pyyaml if Python is used. So every developer that
needs to run 'make .gitlab-ci.yml' and also the docker image used in the
GitLab pipelines would need to have pyyaml pre-installed.
Instead of adding the mentioned dependency, keep using a bash script.

While moving the commands to the script:
 - mimic the behavior of the previous make target and fail on any
   command that fails, by using 'set -e';
 - break the original lines in one command per line, making the diff for
   any patch to be applied to this file to look nicer;
 - keep the script as simple as possible, without functions, just a
   script that executes from the top to bottom;
 - do not perform validations on the input parameters, any command that
   fails already makes the script to fail;
 - do not add an usage message, the script is not intended to be called
   directly.

This patch does not change functionality.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
[Thomas: make the script output on stdout rather than take the output
file name as second argument.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-09 21:30:24 +01:00
Damien Thébault bb7c13273f support/download/dl-wrapper: fix urlencode option never being sent to backend
Since commit 38de434123 ("download: fix file:// BR2_PRIMARY_SITE
(download cache)"), the urlencode option is no longer passed to the
download backend, because we use ${backend} instead of
${backend_urlencode}.

We must get the urlencode information from backend_urlencode.

Signed-off-by: Damien Thébault <damien.thebault@vitec.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
[Thomas: rework commit log]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-09 17:16:04 +01:00
Thomas Petazzoni ba99ccaae4 support/testing: drop docstring in TestPythonPackageBase::test_run()
Having a docstring in the test_run() method clutters the "run-tests
-l" output:

test_run (tests.package.test_python_crossbar.TestPythonPy3Crossbar)
Test a python package. ... ok
[...]
test_run (tests.package.test_python_pexpect.TestPythonPy2Pexpect)
Test a python package. ... ok
test_run (tests.package.test_python_pexpect.TestPythonPy3Pexpect)
Test a python package. ... ok
test_run (tests.package.test_python_twisted.TestPythonPy2Twisted)
Test a python package. ... ok
test_run (tests.package.test_python_twisted.TestPythonPy3Twisted)
Test a python package. ... ok
test_run (tests.package.test_python_pynacl.TestPythonPy2Pynacl)
Test a python package. ... ok
test_run (tests.package.test_python_pynacl.TestPythonPy3Pynacl)
Test a python package. ... ok

So let's simply drop this docstring that is not particularly useful.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-12-08 20:50:08 +01:00
Francois Perrad 9d08630128 support/testing: add prosody test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-06 23:08:59 +01:00
Francois Perrad bcfbf6437a support/testing: add lua test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-06 22:37:10 +01:00
Francois Perrad 2d1f32f084 support/testing: add perl-io-socket-ssl test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-06 22:30:46 +01:00
Francois Perrad 35c858bd82 support/testing: add perl-io-socket-multicast test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-06 22:30:40 +01:00
Francois Perrad d5196b8ee9 support/testing: add perl-encode-decode test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-06 22:30:36 +01:00
Francois Perrad 7046c20476 support/testing: add perl-dbd-mysql test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-06 22:30:30 +01:00
Francois Perrad fe4b91db57 support/testing/tests/packages/test_perl_*: regeneration
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-06 22:19:59 +01:00
Thomas Petazzoni 3e19b837f4 support/scripts/graph-depends: introduce MANDATORY_DEPS array
This array will be re-used in another function in a follow-up commit,
so it makes sense to factor it out.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-06 22:17:07 +01:00
Thomas Petazzoni 659d45adc4 support/scripts/graph-depends: use proper rootpkg in remove_extra_deps()
The remove_extra_deps() function removes dependencies that we are not
interested in seeing in the dependency graph. It does this for all
packages, except the 'all' package, which on full dependency graphs is
the root of the tree.

However, this doesn't take into account package-specific dependency
graphs (i.e make <pkg>-graph-depends) where the root is not 'all', but
'<pkg>'. Due to this, dependencies on "mandatory deps" were not
visible at all, i.e the toolchain package (and its dependencies) and
the skeleton package (and its dependencies) were not displayed in
package-specific dependency graphs.

To fix this, we use the existing rootpkg variable instead of
hardcoding 'all'.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-06 22:16:37 +01:00
Yann E. MORIN 7ee54bd603 support/graph-depends: drop legacy target-exceptions
Currently, we avoid drawing the dependencies that we call 'target
exceptions', becasue they initially were returned by 'show-targets',
when they in fact were not really packages and thus should not be on
the graph.

However, those two exceptions have no longer been reported in the output
of show-targets since we merged very old initial top-level parallel
build way back in 2014, with commit a24877586a (Makefile: add support
for top-level parallel make), where they had been converted into purely
internal rules.

4 years have passed, we can now drop those exceptions from the
graph-depends script.

This concludes the cleanup initiated three years ago with commit
0b32791f00 (graph-depends: remove absent targets from
TARGET_EXCEPTIONS).

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-06 22:15:32 +01:00
Francois Perrad a03f66f431 support/testing: add perl-xml-libxml test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/124872335]
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-03 20:52:15 +01:00
Francois Perrad d9f5748a07 support/testing: add perl-x10 test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/124872334]
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-03 20:52:10 +01:00
Francois Perrad d48dc64c7d support/testing: add perl-mail-dkim test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/124872333]
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-03 20:52:05 +01:00
Francois Perrad 1ac4178f2f support/testing: add perl-libwww-perl test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/124872332]
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-03 20:50:40 +01:00
Francois Perrad 80196db1f4 support/testing: add perl-gdgraph test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/124872330]
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-03 20:50:25 +01:00
Francois Perrad 78f4fad3cf support/testing: add perl-class-load test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/124872329]
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-03 20:50:21 +01:00
Francois Perrad e729bf722b support/testing: add perl test
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-12-03 20:48:49 +01:00
Peter Korsgaard 13c43455a0 Merge branch 'next'
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-12-02 08:16:10 +01:00
Peter Korsgaard 9089a9ff30 Update for 2018.11
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-12-01 23:06:49 +01:00
Erico Nunes 2d9566b7b7 boot/grub2: add option to install tools
Add an option to install grub2 support tools to the target.

In the context of Buildroot, some useful target tools provided are
grub2-editenv, grub2-reboot, which provide means to manage the grub2,
environment, boot order, and others.

Signed-off-by: Erico Nunes <nunes.erico@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-26 20:59:33 +01:00
Thomas Petazzoni eeb6341410 support/scripts/check-host-rpath: split condition on two statements
Inside the check_elf_has_rpath(), we check if the host binary has a
correct RPATH, which should be either an absolute path to
$(HOST_DIR)/lib, or a relative path using $ORIGIN. Those two
conditions are checked in a single statements, but as we are going to
add a third condition, let's split this up a bit:

 - If we have a RPATH to $(HOST_DIR)/lib -> we're good, return 0
 - If we have a RPATH to $ORIGIN/../lib -> we're good, return 0
 - Otherwise, we will exit the loop, and return 1

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-11-26 19:08:57 +01:00
Yann E. MORIN 2218dc85be support/dependencies: add a check for a suitable gzip
Recently, some hash mismatch have been reported, both by users as well
as autobuilder failures, about tarballs generated from git repositories.

This turned out to be caused by users having the 'gzip' command somehow
aliased to 'pigz' (which stand for: parallel implementation of gzip,
which takes advantage of multi-processor system to parallelise the
compression).

Unfortunately, the output of pigz-compressed archives differ from that
of gzip (even though they *are* valid gzip-compressed streams).

Add a dependency check that ensures that gzip is not pigz. If that is
the case, define a conditional dependency to host-gzip, that is used as
a download dependency for packages that will generate compressed files,
i.e. cvs, git, and svn.

Fixes:
    http://autobuild.buildroot.org/results/330/3308271fc641cadb59dbf1b5ee529a84f79e6d5c/

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Marcin Niestrój <m.niestroj@grinn-global.com>
Cc: Erico Nunes <nunes.erico@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-24 15:14:58 +01:00
Yann E. MORIN c3af086395 support/dependencies: treat BSD-tar like the other cases
Currently, when we detect that tar is BSD-tar, we fake an unsupported
version (major, minor) and rely on the version check to reject BSD-tar.

There is no reason to use such shenanigans, when we can simply reject it
from the onset.

Simplify the logic:
  - use positive logic in the condition
  - directly exit in error

Also, comment that case like the other cases are commented.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-24 15:14:50 +01:00
Yann E. MORIN 020206ca57 support/graph-depends: fix package names starting with a non-alpha
Graphviz' dot utility does not like nodes which names does not start
with an ^[[:alpha:]], i.e. 18xx-ti-utils would cause grievance:

    Warning: syntax ambiguity - badly delimited number '18x' in line 4 [...]/graph-depends.dot splits into two tokens
    Warning: syntax ambiguity - badly delimited number '18x' in line 5 [...]/graph-depends.dot splits into two tokens
    Warning: syntax ambiguity - badly delimited number '18x' in line 6 [...]/graph-depends.dot splits into two tokens
    Warning: syntax ambiguity - badly delimited number '18x' in line 7 [...]/graph-depends.dot splits into two tokens

Prefix nodes with an underscore to fix that.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-24 10:58:40 +01:00
Nasser Afshin 7b87c2ae81 support/kconfig/merge_config.sh: avoid false positive matches from comment lines
We are using empty CONFIG_PREFIX_. This results in false positive match
for comment lines when merging config fragments.

To avoid false positive reports, we use separate sed expressions and
address comment lines explicitly.

This is actually is in the Linux kernel mainline (v4.20-rc2):
6bbe4385d035c6fac56f840a59861a0310ce137b
("kconfig: merge_config: avoid false positive matches from comment lines")

Signed-off-by: Nasser Afshin <Afshin.Nasser@gmail.com>
Reviewed-by: Petr Vorel <petr.vorel@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-24 10:11:15 +01:00
Nasser Afshin 8f069a665c support/kconfig/merge_config.sh: fix merging buildroot config files
This patch allows us to define config prefix with CONFIG_ environment
variable.

By setting the proper config prefix, we will have proper 'redundant
configuration warnings' when we use '-r -m' options.

This is actually already in mainline for v4.20-rc1:
2cd3faf87d2d8f6123adf34741b9a7b98828a76f
("merge_config.sh: Allow to define config prefix")

Signed-off-by: Nasser Afshin <afshin.nasser@gmail.com>
Reviewed-by: Petr Vorel <petr.vorel@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-24 10:11:11 +01:00
Ricardo Martincoski bac9a78646 support/testing: add python-ubjson tests
Add a simple test case to check the basic usage, storing a dict into a
file and then retrieving the dict from the file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-23 22:05:46 +01:00
Ricardo Martincoski 2979ab9bd5 support/testing: add python-treq tests
Use a simple script to check the basic usage. The target has no https
server, so a connection from in the target to localhost must not
succeed.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-23 22:05:46 +01:00
Ricardo Martincoski ac010beec5 support/testing: add python-subprocess32 test
Add a simple test case to check the basic usage by calling 'ls' and
checking the output.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-23 22:05:46 +01:00
Ricardo Martincoski d144edb21d support/testing: add python-service-identity tests
Add a simple test case that imports the module.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-23 22:05:46 +01:00
Ricardo Martincoski 74d61681f1 support/testing: add python-pyyaml tests
Add a simple test case to check the basic usage, storing a dict into a
file and then retrieving the dict from the file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-23 22:05:46 +01:00
Ricardo Martincoski 7e69ddc9e7 support/testing: add python-pynacl tests
Add a simple test case that minimally uses the module.
Add haveged to the target to generate enough entropy so pynacl ->
libsodium don't hang waiting for /dev/random.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-23 22:05:46 +01:00
Ricardo Martincoski cdad4bf6aa support/testing: add python-pexpect tests
Add a simple test case to check the basic usage. Call 'login' and try
wrong user/password, expecting the 'Login incorrect' message.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-23 22:05:46 +01:00
Ricardo Martincoski 756dd5f132 support/testing: add python-passlib tests
Add a simple test case that creates a hash for a password and verifies
it against an incorrect and a correct password.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-23 22:05:46 +01:00
Ricardo Martincoski 9ee9566640 support/testing: add python-constantly tests
Add a simple test case to check the basic usage by creating a class with
two constants.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-23 22:05:46 +01:00
Ricardo Martincoski 887248d354 support/testing: add python-click tests
Use a simple script to check the basic usage. Since this package
provides command line arguments, override run_sample_scripts to call the
script with arguments and check the expected output.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-23 22:05:46 +01:00
Ricardo Martincoski 61b4b81c44 support/testing: add python-cbor tests
Add a simple test case to check the basic usage, storing a dict into a
file and then retrieving the dict from the file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-23 22:05:46 +01:00
Ricardo Martincoski 38557a8c63 support/testing: add python-bitstring tests
Add a simple test case to check the basic usage by checking the
corresponding representation of a 12-bit decimal number in hex, binary
and integer.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-23 22:05:46 +01:00
Ricardo Martincoski 66f99555ed support/testing: add python-automat tests
Use a minimal script to check the basic usage by creating and using a
small state machine.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-23 22:05:46 +01:00
Ricardo Martincoski 367c045512 support/testing: add python-attrs tests
Use a minimal script to check the basic usage creating a class with 2
attributes.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-23 22:05:46 +01:00
Ricardo Martincoski 1bd9e9c199 support/testing: add python-argh tests
Use a simple script to check the basic usage. Since this package
provides command line arguments, override run_sample_scripts to call the
script with arguments and check the expected output.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-23 22:05:46 +01:00
Ricardo Martincoski adec30e4f0 support/testing/test_rust: use standard defconfig fragment style
Since commit "2927f412be support/testing: standardize defconfig
fragments style" all other test cases use the same style for defconfig
fragments:
 - start after a backslash;
 - be declared as a multi-line string literal;
 - be indented one level more than the variable that contains it.

Do the same here for consistency.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Matt Weber <matthew.weber@rockwellcollins.com>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-20 09:01:08 +01:00
Peter Korsgaard 446a6f91b2 make: configure host-make with host- prefix
Fixes:
http://autobuild.buildroot.net/results/e29/e293aadc692d2ed337881ef2172ddf66a60bc05c/

And many more.

Install as 'host-make' rather than just 'make', as that otherwise confuses a
number of packages when they invoke recursive / sub-make.  The internal job
control logic of GNU make is version dependant, so mixing versions may lead
to issues like:

make[1]: Entering directory `/home/peko/autobuild/instance-0/output/build/boa-0.94.14rc21'
(cd src && make -w --jobserver-fds=5,6 -j)
make: unrecognized option '--jobserver-fds=5,6'

With this rename, only packages explicitly opting in for our host-make
(using the BR2_MAKE / BR2_MAKE_HOST_DEPENDENCY logic) will use it.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-11-20 08:17:43 +01:00
Yegor Yefremov ad361c69f6 support/testing: add python-crossbar tests
This test invokes "crossbar version" command, that checks all
dependencies found in setup.py files and prints some system related
information.
Add haveged to the target to generate enough entropy so crossbar ->
pynacl -> libsodium don't hang waiting for /dev/random.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
[Ricardo: move test script to a separate file, remove Python 2 variant,
 add haveged to target to add entropy and avoid hanging]
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-13 20:55:49 +01:00
Ricardo Martincoski 6a24761c72 support/testing: use TestPythonPackageBase for python-txtorcon
Move the test script to be run on the target from inline in the test
case to a separate file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-13 20:55:45 +01:00
Ricardo Martincoski ac4a052968 support/testing: use TestPythonPackageBase for python-txaio
Move the test scripts to be run on the target from inline in the test
case to a separate file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-13 20:55:45 +01:00
Ricardo Martincoski 0f98e58cac support/testing: use TestPythonPackageBase for python-twisted
Move the test script to be run on the target from inline in the test
case to a separate file.

Get the base defconfig fragment from the immediate parent class and not
directly from TestPythonBase because it is the correct way of doing
this. This way the base class TestPythonTwisted could even be placed in
a separate file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-13 20:55:45 +01:00
Ricardo Martincoski ee6b37cf87 support/testing: use TestPythonPackageBase for python-incremental
Move the test script to be run on the target from inline in the test
case to a separate file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-13 20:55:45 +01:00
Ricardo Martincoski f07994f7d6 support/testing: use TestPythonPackageBase for python-cryptography
Move the test script to be run on the target from inline in the test
case to a separate file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-13 20:55:45 +01:00
Ricardo Martincoski b5dd9364ee support/testing: use TestPythonPackageBase for python-autobahn
Move the test script to be run on the target from inline in the test
case to a separate file.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-13 20:55:45 +01:00
Ricardo Martincoski 13e3c14ed2 support/testing: create default test case for python packages
Test cases for python packages are very similar among each other: run a
simple script in the target that minimally tests the package.
So create a new helper class named TestPythonPackageBase that holds all
the logic to run a script on the target.

TestPythonPackageBase adds in build time one or more sample scripts to
be run on the target. The test case for the python package must
explicitly list them in the "sample_scripts" property. The test case
then automatically logins to the target, checks the scripts are really
in the rootfs (it calls "md5sum" instead of "ls" or "test" in an attempt
to make the logfile more friendly, since someone analysing a failure can
easily check the expected script was executed) and then calls the python
interpreter passing the sample script as parameter.
An optional property "timeout" exists for the case the sample script
needs more time to run than the default timeout from the test infra
(currently 5 seconds).

A simple test case for a package that only supports Python 2 will look
like this:

|from tests.package.test_python import TestPythonPackageBase
|
|
|class TestPythonPy2<Package>(TestPythonPackageBase):
|    __test__ = True
|    config = TestPythonPackageBase.config + \
|        """
|        BR2_PACKAGE_PYTHON=y
|        BR2_PACKAGE_PYTHON_<PACKAGE>=y
|        """
|    sample_scripts = ["tests/package/sample_python_<package>.py"]
|    timeout = 15

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-13 20:55:45 +01:00
Yann E. MORIN 409d4c3fe9 fs: don't use an intermediate tarball
Since 118534fe54 (fs: use a common tarball as base for the other
filesystems), the filesystem creation is split in two steps, using an
intermediate tarball to carry the generic, common finalisations to the
per-filesystem finalisation and image creation.

However, this intermediate tarball causes an issue with capabilities:
they are entirely missing in the generated filesystems.

Capabilities are stored in the extended attribute security.capability,
which tar by default will not store/restore, unless explicitly told to,
e.g. with --xattrs-include='*', which we don't pass.

Now, passing this option when creating and extracting the intermediate
tarball, both done under fakeroot, will cause fakeroot to report an
invalid filetype for files with capabilities. mksquashfs would report
such unknown files as a warning, while mkfs.ext2 would fail (with a
similar error message), e.g.:

    File [...]/usr/sbin/getcap has unrecognised filetype 0, ignoring

This is due to a poor interaction between tar and fakeroot; running as
root the exact same commands we run under fakeroot, works as expected.
Unfortunately, short of fixing fakeroot (which would first require
understanding the problem in there), we don't have much options.

The intermediate tarball was made to avoid redoing the same actions over
and over again for each filesystem to build. However, most of the time,
only one or two such filesystems would be enabled [0], and those actions
are usually pretty lightweight. So, using an intermediate tarball does
not provide a big optimisation.

The main reason to introduce the intermediate tarball, however, is that
it allows to postpone per-filesystem finalisations to be applied only
for the corresponding filesystem, not for all of them.

So, we get rid of the intermediate tarball, and simply move all of the
code to run under fakeroot to the per-filesystem fakeroot script.
Instead of extracting the intermediate tarball, we just rsync the
original target/ directory, and apply the filesystem finalisations on
that copy. The only thing still done in the rootfs-common step is to
generate the intermediate files (users file, devices file) that are used
in the fakeroot script.

Fixes: https://bugs.busybox.net/show_bug.cgi?id=11216

Note: an alternate solution would have been to keep the intermediate
tarball to keep most of the common finalisations, and move only the
permissions to each filesystem, but that was getting a bit more complex
and changed the ordering of permissions and post-fakeroot scripts. Once
we bite the bullet of having some common finalisation done in each
filesystem, it's easier to just move all of them.

[0] Most probsably, users would enable the real filesystem to put on
their device, plus the 'tar' filesystem, to be able to easily inspect
the content on their development machine.

Reported-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2018-11-13 00:47:36 +01:00
Thomas Petazzoni bff6b61adf support/testing/tests/fs/test_f2fs: fix flake8 warning
This commit fixes the following flake8 warnings:

support/testing/tests/fs/test_f2fs.py:6:1: E302 expected 2 blank lines, found 1
support/testing/tests/fs/test_f2fs.py:12:1: E302 expected 2 blank lines, found 1
support/testing/tests/fs/test_f2fs.py:38:23: E225 missing whitespace around operator

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-08 22:41:53 +01:00
Grzegorz Blach 5244fd29a1 support/testing: add test for the f2fs filesystem support
Signed-off-by: Grzegorz Blach <grzegorz@blach.pl>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-08 21:17:51 +01:00
Serj Kalichev c77cd17082 support/scripts/mkmakefile: make wrapper silent by default
Suppose we use Makefile wrapper and build some project out of
buildroot tree (O=...). A command like "make
busybox-all-external-deps" will output the string "uname 022 && make
..." to stdout before the usefull information. It pollutes stdout. At
the same time if we use the same command in the buildroot source-tree
then we don't get the additional output. This patch makes wrapper
silent by default. People who prefer to see more verbose output can
use V=1.

Signed-off-by: Serj Kalichev <serj.kalichev@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2018-11-02 21:23:02 +01:00
Petr Vorel 0d61e067eb kconfig: Refresh patches
Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-10-30 20:58:54 +01:00
Petr Vorel a3366b2705 kconfig: Restore missing custom changes + add patches
Commit 6eacea5ae0 accidentally removed these changes in merge_config.sh:
0f56304521 ("merge_config.sh: create temporary files in /tmp")
28fac3973b ("merge_config.sh: add br2-external support")

Changes were lost because commits just changed files, but didn't add patches.
Therefore not only restore our changes, but also add (updated) patches.

Missing 0f56304521 caused breaking merge_config.sh when used in out of
tree build:
$ make -C buildroot O=$PWD/output defconfig
...
$ cd output
$ echo 'BR2_TARGET_GENERIC_HOSTNAME="test"' > test.frag
$ ../buildroot/support/kconfig/merge_config.sh .config test.frag
Using .config as base
Merging test.frag
umask 0022 && make -C /home/test/buildroot O=/home/test/output/. alldefconfig
  GEN     /home/test/output/Makefile
*** Can't read seed configuration "./.tmp.config.qIcpASpUyh"!
make[1]: *** [Makefile:925: alldefconfig] Error 1
make: *** [Makefile:16: _all] Error 2

Fixes: 6eacea5ae0 support/kconfig: bump to kconfig from Linux 4.17-rc2

Signed-off-by: Petr Vorel <petr.vorel@gmail.com>
Reported-by: Erico Nunes <nunes.erico@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2018-10-30 20:58:09 +01:00