One of the possible usages of check-package is to first cd to the
directory that contains the files to test (e.g. a package directory) and
then call the script passing the files in the current dir.
It already works when used for intree files, but for files in a
br2-external it throws an exception because some check functions (from
utils/checkpackagelib/lib_*.py) do need the name of the file being
processed and assume there will be a slash before the name.
Fix all check functions that assume that the full filename being checked
contains a slash. Do not use regexps to extract the filename, use
os.path functions instead.
Notice RemoveDefaultPackageSourceVariable and TypoInPackageVariable lead
to an exception in this case, but ApplyOrder instead generates a false
warning.
Fixes bug #11271.
Reported-by: Vitaliy Lotorev <lotorev@gmail.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Vitaliy Lotorev <lotorev@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Fix an issue introduced by Arnout while committing. Jerzy originally
initialized the menu_of_packages, package and print_package_warning
members like they should be, but Arnout thought it wasn't needed and
removed that.
It is actually needed, to make sure the top level (level 0) works.
Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/264383157
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The 'source' strings identify which package is incorrectly ordered. We
need to extract the actual package name from that string, which is
currently done with constants that assume the file is package/Config.in.
In addition, only 'source' lines that are indented with a tab are
checked. This kind of indentation is done in package/Config.in, but not
e.g. boot/Config.in.
Therefore, use a regular expression to match the 'source' lines, and to
extract the directory part from it.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In the future, the nesting level of menus, comments and conditions may
increase. The fixed array length used now is not appropriate. Therefore,
append elements to the arrays if needed.
Also change order of variables.
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The CommentsMenusPackagesOrder check builds the 'state' to track the
depth of menus and conditions. However, a menuconfig doesn't create a
menu by itself - it is always followed by a condition that implies the
menu. As a result, when unwinding the 'state', the level will be wrong.
Fix this by checking for menu followed by a space, so it no longer
matches menuconfig. For consistency, do the same for comment and if
as well.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The CommentsMenusPackagesOrder test is broken in various ways for files
other than package/Config.in and package/Config.in.host. Therefore, the
script gives bogus errors for various other Config.in files.
However, we don't really want to check those other files. Indeed, many
of them have a non-alphabetical ordering for good reasons.
Therefore, skip the check for files other than package/Config.in and
package/Config.in.host.
Fixes: https://gitlab.com/buildroot.org/buildroot/-/jobs/251214899
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Jerzy Grzegorek <jerzy.m.grzegorek@gmail.com>
[Arnout:
- calculate level by counting - instead of with a static array;
- new_package is only used locally, so don't make it a class member;
- do indentation according to length of prefix;
- don't split string in the middle of a line;
- report first wrong package per menu;
- do replace() only once;
- add comment why we do replace().
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
When running in a CI system, stat messages become white noise. Introduce
an option to suppress non-error, non-warning, messages.
Signed-off-by: Brandon Maier <brandon.maier@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
test-pkg will use gigabytes of space when testing all toolchains.
Nevertheless, you are normally only interested in the actual build / host
tree when there is a build failure.
Do a 'make clean' for successful builds to save disk space, unless the new
option '-k/--keep' is set.
Note that the logfile and configuration is always retained for inspection.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The long option parsing of test-pkg is broken because:
- some long options are not declared
- there should be a comma between long options, the colon does not replace
it.
This change also revealed that the declaration of 'toolchains-dir' should
have been 'toolchains-csv', originally introduced in commit ed59f81a3c.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Acked-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Initial patch to start reproducibility testing.
Also enable tar filesystem to compare images since diffoscope works well
with tar.
Signed-off-by: Atharva Lele <itsatharva@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Instead of stdout where it gets mixed with the normal output, confusing
software parsing the output (E.G. get-developers -e as git sendemail.ccCmd).
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The XS modules have a high propability to fail (compared to Pure Perl modules),
so it is valuable to check XS dependencies before the check of the main module.
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
It is possible to generate one-line config for the package just by
normalize it to the form:
BR2_PACKAGE_${pkg_replaced-to_and_uppercase}
it simplifes a bit of testing package where no additional config options
are needed.
Signed-off-by: Vadim Kochan <vadim4j@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Some python packages seem to use underscores in inconsistent ways. We can
attempt to normalize these by always using dashes for the buildroot name and
attempting to autodetect the correct metadata name format.
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
For details, see https://github.com/snyk/zip-slip-vulnerability
Older python versions do not validate that the extracted files are inside
the target directory. Detect and error out on evil paths before extracting
.zip / .tar file.
Given the scope of this (zip issue was fixed in python 2.7.4, released
2013-04-06, scanpypi is only used by a developer when adding a new python
package), the security impact is fairly minimal, but it is good to get it
fixed anyway.
Reported-by: Bas van Schaik <security-reports@semmle.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Now that Qt4 is removed, these fixups can be removed as well.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
For the general case, appending values to variables is OK and also a
good practice, like this:
|PACKAGE_VAR = value1
|ifeq ...
|PACKAGE_VAR += value2
or this, when the above is not possible:
|PACKAGE_VAR = value1
|ifeq ...
|PACKAGE_VAR := $(PACKAGE_VAR), value2
But this override is an error:
|PACKAGE_VAR = value1
|PACKAGE_VAR = value2
as well this one:
|ifeq ...
|PACKAGE_VAR += value1
|endif
|PACKAGE_VAR = value2
And this override is error-prone:
|PACKAGE_VAR = value1
|ifeq ...
|PACKAGE_VAR = value2
Create a check function to warn about overridden variables.
Some variables are likely to have a default value that gets overridden
in a conditional, so ignore them. The name of such variables end in
_ARCH, _CPU, _SITE, _SOURCE or _VERSION.
After ignoring these variable names, there are a few exceptions to this
rule in the tree. For them use the comment that disables the check.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Simon Dawson <spdawson@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Titouan Christophe <titouan.christophe@railnova.eu>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently check-package only knows about ifeq/ifneq.
Add code to handle ifdef/ifndef as well.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently any exceptions for a check function need to be coded into the
check-package script itself.
Create a pattern that can be used in a comment to make check-package
ignore one or more warning types in the line immediately below:
# check-package Indent, VariableWithBraces
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This script currently uses "/usr/bin/env python" as shebang but it does
not really support Python3. Instead of limiting the script to Python2,
fix it to support both versions.
So change all imports to absolute imports because Python3 follows PEP328
and dropped implicit relative imports.
In order to avoid errors when decoding files with the default 'utf-8'
codec, use errors="surrogateescape" when opening files, the docs for
open() states: "This is useful for processing files in an unknown
encoding.". This argument is not compatible with Python2 open() so
import 'six' to use it only when running in Python3.
As a consequence the file handler becomes explicit, so use it to close()
the file after it got processed.
This "surrogateescape" is a simple alternative to the complete solution
of opening files with "rb" and changing all functions in the lib*.py
files to use bytes objects instead of strings. The only case we can have
non-ascii/non-utf-8 files being checked by the script are for patch
files when the upstream file to be patched is not ascii or utf-8. There
is currently one case in the tree:
package/urg/0002-urg-gcc6-fix-narrowing-conversion.patch.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Tested-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Commit 62d5558f76 (utils/get-developers: make it callable from elsewhere
than the toplevel directory) tried to fix this by passing in the toplevel
directory when the DEVELOPERS file is parsed.
Unfortunately this is not enough, as E.G. also the paths listed in the
patches are relative to the toplevel directory, causing it to not match the
entries in the DEVELOPERS file.
In concept this can be fixed by also passing the toplevel directory to the
Developers class, but the simplest solution is just to chdir to the toplevel
Buildroot directory before calling any of the getdeveloperlib functions.
This does require us to finish parsing command line arguments (which opens
the provided patch files) to not get into trouble with relative paths to
patches before chdir'ing / initializing getdeveloperlib.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This reverts commit 62d5558f76.
This actually does not work, as patches contain paths relative to the
toplevel directory as well.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Currently, if a user runs "make" while specifying a specific package
(IE: make -p foo), the Makefile logic skips checking to see if all the
dependencies are selected in the specified packages config file. This behavior
is useful to test simple packages which do not have "complex" dependencies.
However; if a developer uses test-pkg -p ${package_name} to check their package,
the package may pass all the checks, but would have otherwise failed with a
simple "make" because the developer may have failed to add a select line in
packages config file, even if there is a new dependency in the packages
Makefile.
Pass the environment variable "BR_FORCE_CHECK_DEPENDENCIES" to the Makefile in
the test-pkg script, and check it's value in the Makefile. If the value is
"YES" force checking for dependency issues.
Signed-off-by: Adam Duskett <Aduskett@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
For some packages, there's no need to add the _SOURCE variable, since
the name of the source file is the same as the name of the package
(like python-engineio). Hence, we'll add it to the .mk file only if
needed.
Signed-off-by: Asaf Kahlon <asafka7@gmail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Move imports from six package after the standard modules. Resolves
pylint warnings.
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently all random defconfigs which are used in autobuilder use size
optimizaion (-Os), since BR2_OPTIMIZE_S=y is the default.
Adding "-O2" optimization will give better test coverage.
In many cases software gets built with speed optimization rather than
size optimization. So let's add Level 2 optimizaion option to be
generated in random defconfigs, so we could be able to test how
packages are built with "-O2" in autobuilder.
Signed-off-by: Evgeniy Didin <Evgeniy.Didin@synopsys.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: arc-buildroot@synopsys.com
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This commit extends the scancpan script to automatically generate a
test for the Perl module, either if the Perl module uses native
library, or if it has more than one dependency.
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
We should use an empty prefix as we do not have any prefix.
Note that BR2_ is mere a convention.
Signed-off-by: Nasser Afshin <Afshin.Nasser@gmail.com>
Reviewed-by: Petr Vorel <petr.vorel@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When adding the custom BR2_WGET value in the configuration,
genrandconfig forgets to add a newline. Due to this, the next option
that is added is printed on the same line as BR2_WGET="", which causes
it to be ignored.
Due to this, in all builds, the line right after BR2_WGET was
ignored. It could have been BR2_ENABLE_DEBUG, BR2_INIT_BUSYBOX,
BR2_INIT_SYSTEMD, BR2_ROOTFS_DEVICE_CREATION_DYNAMIC_EUDEV,
BR2_STATIC_LIBS or BR2_PACKAGE_PYTHON_PY_ONLY depending on the
randomization.
Fix that by adding a proper newline at the end of the BR2_WGET option.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Some packages have archive name that is different from package name.
For example websocket-client's archive name is websocket_client-*.tar.gz.
scanpypi expects the temporary extract folder to be:
/tmp-folder/BR-package-name/PyPI-packagename-and-version
In the case of websocket-client package the real extraction folder
will be different from the expected one because of the '_' in the
archive file name.
Use archive file name instead of package name to specify the extraction
folder. As the version is already part of this file, we don't need to
specify it.
Bonus: remove obsolete "return None, None" as the function doesn't return
anything. OSError class doesn't provide "message" member, so replace it
with "strerror".
Fixes:
https://bugs.busybox.net/show_bug.cgi?id=11251
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Reviewed-by: Asaf Kahlon <asafka7@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, utils/scancpan always outputs what should be placed in
package/Config.in to include all Perl packages Config.in
files. However, in practice, this is only useful when a new package is
added. This commit adjusts this behavior so that what should be place
in package/Config for Perl packages is only displayed when scancpan
has produced a new Buildroot package for a Perl module.
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
This variable is needed for upgrading packages with scancpan.
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The utils/diffconfig script works only on variables with the BR2_
prefix. This is OK for Buildroot [def]configs since this is the prefix
for all user-facing variables, but it prevents using the same script
to compare configs from kconfig-based packages.
Remove the BR2_ restriction, allowing usage such as:
./utils/diffconfig \
board/qemu/xtensa-lx60/linux.config \
board/qemu/xtensa-lx60/linux-nommu.config
Signed-off-by: Marcel Patzlaff <m.patzlaff@pilz.de>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
there are also runtime dependency
Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
get-developers tries to open DEVELOPERS in the current directory, so it
breaks when calling it from elsewhere than the toplevel Buildroot directory.
Traceback (most recent call last):
File "../utils/get-developers", line 107, in <module>
__main__()
File "../utils/get-developers", line 26, in __main__
devs = getdeveloperlib.parse_developers(os.path.dirname()
File "/home/peko/source/buildroot/utils/getdeveloperlib.py", line 161, in parse_developers
with open(os.path.join(basepath, "DEVELOPERS"), "r") as f:
IOError: [Errno 2] No such file or directory: '/home/peko/source/buildroot/output-foo/DEVELOPERS'
Fix it by instead figuring out where the DEVELOPERS file is relative to the
location of get-developers (E.G. one level up).
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
[Arnout:
- add realpath to support a symlinked get-developers script;
- pass devs_dir argument to check_developers() to support -c in subdir;
- convert basepath to absolute path to support -f option.
]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
This fixes a syntax error introduced in bcf2ed5cc3.
Output before the patch:
$ ./utils/get-developers outgoing/*
File "./utils/get-developers", line 97
print dev
^
SyntaxError: Missing parentheses in call to 'print'. Did you mean
print(dev)?
Output after the patch:
$ ./utils/get-developers outgoing/*
git send-email --to buildroot@buildroot.org
Signed-off-by: Grégoire Delattre <gregoire.delattre@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When used without spdx_lookup the BSD licence cannot be
detected correctly because many Python packages just specify
BSD without the exact version in their metadata. So add a
special message warning the user instead of the licence id.
Bonus: fix typo.
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
