We need to update the sysconfigdata path for python 3.8.0 per upstream
commit 7efc526e5cfb929a79c192ac2dcf7eb78d3a4401 which indicates the "m"
flag has been removed from sys.abiflags.
Fixes this build error when building target packages:
ValueError: Empty module name
Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
No functional change is brought by this modification.
This patch:
* removes redundant <pkg>_SRCDIR and <pkg>_BUILDDIR declarations
(already defined in pkg-generic.mk)
* documents the usage of <pkg>_SUBDIR in the python-specific section of
the manual.
Signed-off-by: Lionel Orry <lionel.orry@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, host Python packages that use setuptools are installed as
Python Eggs, i.e they are installed in
$(HOST_DIR)/lib/python2.7/site-packages/Something-<version>.egg. Once
installed, each Python Egg is registered to a file called
$(HOST_DIR)/lib/python2.7/site-packages/easy-install.pth. This file is
read by the Python interpreter so that the installation location of
each Egg is added to the Python path, and can be found by the Python
interpreter.
However, the fact that the installation of different Python modules
need to update a common file is clearly not compatible with
per-package directories and top-level parallel build.
To fix this, we avoid using Python Eggs using the same
--single-version-externally-managed option that we use for target
modules. This option is normally meant for distributions packaging
Python modules, and can therefore only be used if either --record (to
record the list of files being installed) or --root is
passed. --root=/ works fine and was suggested by
https://stackoverflow.com/questions/6301003/stopping-setup-py-from-installing-as-egg/33791008#33791008.
With this change, host Python modules installed by setuptools are now
installed in the "regular" way, i.e directly in
$(HOST_DIR)/lib/python2.7/site-packages/mako for host-python-mako.
This makes the installation of host Python modules more similar to the
one of target modules, and makes it compatible with per-package
directory support and top-level parallel build.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Asaf Kahlon <asafka7@gmail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When a package uses "setuptools" as its <pkg>_SETUP_TYPE, we currently
add a dependency on host-python-setuptools. This means that:
(1) When BR2_PACKAGE_PYTHON=y, the default host Python version is
Python 2.x, and host-python-setuptools is installed for
host-python.
(2) When BR2_PACKAGE_PYTHON3=y, the default host Python version is
Python 3.x, and host-python-setuptools is installed for
host-python3.
(3) When no target Python interpreter is selected, the default host
Python version is Python 2.x, and host-python-setuptools is
installed for host-python.
Situations (1) and (3) are problematic for host Python packages that
need Python 3.x. Such packages use <pkg>_NEEDS_HOST_PYTHON = python3,
but if they use setuptools as their setup type, they will not find
setuptools installed for host-python3 in situations (1) and (3)
described above.
We currently have a single package that sets <pkg>_NEEDS_HOST_PYTHON =
python3: host-meson. host-meson generally works because if setuptools
is not found, it falls back to distutils, which is part of the
standard Python library. However, if there is a setuptools version
installed system-wide, it may be picked up, but may not necessarily be
the same version as Buildroot setuptools, potentially causing
problems.
This commit makes the necessary change to the python-package
infrastructure to fix this behavior, by identifying the following
cases:
- When a host Python package says <pkg>_NEEDS_HOST_PYTHON = python3,
then we know it wants setuptools installed for host-python3, so we
use host-python3-setuptools.
- When a host Python package says <pkg>_NEEDS_HOST_PYTHON = python2,
then we known it wants setuptools installed for host-python, so we
use host-python-setuptools.
- When BR2_PACKAGE_PYTHON3=y, and we have a target package, or a host
package with no NEEDS_HOST_PYTHON option, then we want setuptools
installed for host-python3, so we use host-python3-setuptools.
- When BR2_PACKAGE_PYTHON=y or no target interpreter is enabled at
all, and we have a target package, or a host package with no
NEEDS_HOST_PYTHON option, then we want setuptools for host-python,
so we use host-python-setuptools.
To make this happen, we use host-python3-setuptools introduced in a
previous commit, but we also change host-python-setuptools to force
its installation for host-python. The latter is needed if you build
with BR2_PACKAGE_PYTHON3=y but want to install a Python-based package
that has NEEDS_HOST_PYTHON=python2.
There is one single package that needs be adjusted following this:
lirc-tools, because it is not using the python-package
infrastructure. It directly depends on host-python-setuptools, which
no longer works because host-python-setuptools now only installs for
Python 2.x, while lirc-tools Python binding only supports Python
3.x. Switching to host-python3-setuptools solves this problem.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Asaf Kahlon <asafka7@gmail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Some python software refers to setuptool/distutils options
to install files in python root directory (like data_files option).
To use this type of option, python root should point to the real python
root in buildroot folder and not to the guest os /.
Prefix path is always built starting from the python root, so it should
be simply /usr.
Signed-off-by: Angelo Compagnucci <angelo@amarulasolutions.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Since things are no longer installed in $(HOST_DIR)/usr, the callers
should also not refer to it.
This is a mechanical change with
git grep -l '$(HOST_DIR)/usr/bin' | xargs sed -i 's%$(HOST_DIR)/usr/bin%$(HOST_DIR)/bin%g'
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Remove the redundant usr/ component of the HOST_DIR paths. Since a
previous commit added a symlink from $(HOST_DIR)/usr to $(HOST_DIR),
everything keeps on working.
Build-tested with a bunch of python packages.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
According to PEP 370 Python will also search for the packages in the
user site-packages directory. This can affect build reproducibility.
The solution is to use PYTHONNOUSERSITE=1 for all Python packages,
i.e. both host and target variants.
Fixes bug #9791.
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Update all appropriate version numbers as well as SHAs and MD5s as well
as repbasing BR's patches on top of 3.6.1 codebase (new github repo
tree, v3.6.1 tag was used). Note that patch:
[PATCH] Change the install location of _sysconfigdata.py
was dropped due to the fact taht build system now adds platform
specific suffix to sysconfigdata's name, so each platform's file
should have a unique name and distutils now allows to specify which
sysconfigdata is used via _PYTHON_SYSCONFIGDATA_NAME
see:
c4b53afce492dec548ff
and patches:
[PATCH] distutils/sysconfig: use sysconfigdata
[PATCH] setup.py: do not add invalid header locations
[PATCH] Do not harcode invalid path to ncursesw headers
was dropped since it looks like it made it's way upstream, see:
409482251b1351c31aa9e13c3201fb
respectively.
Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
This feature consists in automatically deducing dependencies of a host
package from the dependencies of the target variant.
However, it causes some issues, and many host packages need different
dependencies than their target variants.
Now that host dependencies are explicitly set for all packages, we can
remove it.
Signed-off-by: Julien Floret <julien.floret@6wind.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Currently, each python package (be it the python interpreter package
itself or external python modules) is responsible for compiling its
.py into .pyc files. Unfortunately, this is not ideal as some packages
only install .py files without compiling them into .pyc files. In this
case, if the Buildroot configuration specifies to keep only the .pyc
files, the .py files are removed and lost.
To address this, this commit changes the logic by making the
compilation of .pyc files a global operation: the python interpreter
packages register a target finalize hook that is in charge of
compiling all installed .py files.
The *.pyc generation on a per package basis is disabled in the
python-package infrastructure by passing the "--no-compile" option to
setup.py.
The *.pyc generation for the Python interpreter internal modules is
disabled through --disable-pyc-build configure option.
A small helper script is used to perform the compilation, the purpose
of this script is to abort the compilation process if one of the .py
file cannot be compiled. It has been provided by Samuel Martin and
integrated into this commit.
Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
Cc: Samuel Martin <s.martin49@gmail.com>
[Thomas:
- rework for python 3.5
- integrate Samuel proposal that allows to detect compilation
failures.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Create a proper package for the skeleton.
The main Makefile is modified to remove the skeleton support.
The 'dirs' target, will create the $(TARGET_DIR).
The file 'output/target/.root' doesn't exists anymore, as there's no
Make rule to statisfy.
The infrastructure are modified to filter host-skeleton.
It's needed becauses the host-dependencies are derived from the
dependencies of the target package where 'host-' is preprended to the
depedency name.
In the pkg-generic we add skeleton as a dependency to every package.
The whole system/system.mk is now removed at the profit of
package/skeleton/skeleton.mk
[Thomas:
- rebase on top of master and fix some minor conflicts
- remove the 'select BR2_PACKAGE_SKELETON' in
BR2_ROOTFS_SKELETON_DEFAULT and BR2_ROOTFS_SKELETON_CUSTOM, since
anyway the skeleton package is always enabled.
- fixup a few mistakes in the getty handling due to misnamed
variables.]
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
To be consistent with the recent change of FOO_MAKE_OPT into FOO_MAKE_OPTS,
make the same change for FOO_BUILD_OPT.
Sed command used:
find * -type f | xargs sed -i 's#_BUILD_OPT\>#&S#g'
Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
To be consistent with the recent change of FOO_MAKE_OPT into FOO_MAKE_OPTS,
make the same change for FOO_INSTALL_STAGING_OPT.
Sed command used:
find * -type f | xargs sed -i 's#_INSTALL_STAGING_OPT\>#&S#g'
Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
To be consistent with the recent change of FOO_MAKE_OPT into FOO_MAKE_OPTS,
make the same change for FOO_INSTALL_TARGET_OPT.
Sed command used:
find * -type f | xargs sed -i 's#_INSTALL_TARGET_OPT\>#&S#g'
Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
To be consistent with the recent change of FOO_MAKE_OPT into FOO_MAKE_OPTS,
make the same change for FOO_INSTALL_OPT.
Sed command used:
find * -type f | xargs sed -i 's#_INSTALL_OPT\>#&S#g'
Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
In English, unlike in French, almost all usages of the word 'information'
are uncountable, meaning that 'informations' is invalid.
This patch fixes this typo throughout the tree, except in CHANGES and
docs/news.html (historic text).
Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
In English, unlike in French, the 'h' in 'host' is pronounced, meaning that
the article should be 'a' instead of 'an'.
Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
python-numpy needs to be installed to the staging directory, since it
also installs some header files. Therefore, this commit extends the
Python package infrastructure to support staging installation.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
The inner-xxx-targets in the buildroot package infrastructures are
evaluated using $(eval) which causes variable references to be a bit
different than in regular make code. As we want most references to be
expanded only at the time of the $(eval) we should not use standard
references $(VAR) but rather use double dollar signs $$(VAR). This includes
function references like $(call), $(subst), etc. The only exception is the
reference to pkgdir/pkgname and numbered variables, which are parameters to
the inner block: $(1), $(2), etc.
This patch introduces consistent usage of double-dollar signs throughout the
different inner-xxx-targets blocks.
In some cases, this would potentially cause circular references, in
particular when the value of HOST_FOO_VAR would be obtained from the
corresponding FOO_VAR if HOST_FOO_VAR is not defined. In these cases, a test
is added to check for a host package (the only case where such constructions
are relevant; these are not circular).
Benefits of these changes are:
- behavior of variables is now again as expected. For example, setting
$(2)_VERSION = virtual in pkg-virtual.mk will effectively work, while
originally it would cause very odd results.
- The output of 'make printvars' is now much more useful. This target shows
the value of all variables, and the expression that led to that value.
However, if the expression was coming from an inner-xxx-targets block, and
was using single dollar signs, it would show in printvars as
VAR = value (value)
while if double dollar signs are used, it would effectively look like
VAR = value (actual expression)
as is intended.
This improvement is for example effective for FOO_DL_VERSION, FOO_RAWNAME,
FOO_SITE_METHOD and FOO_MAKE.
The correctness of this patch has been verified using 'make printvars',
'make manual' and 'make legal-info' before and after applying this patch,
and comparing the output.
Insight-provided-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Thanks to the 2 previous patches of the series, BR_PATH contains
all locations in which host-packages may install programs.
This patch replaces the occurrences TARGET_PATH and HOST_PATH with
BR_PATH, everywhere these variables are used in the *.mk files.
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Some packages need a host-python interpreter with a version different
from the one installed in the target to run some build scripts (eg.
scons requires python2 to run, to build any kind of packages even if
the python interpreter selected for the target is python3).
In such cases, we need to add the right host-python dependency to the
package using the host-python-package infrastructure, and we also want
to invoke the right host python interpreter during the build steps.
This patch adds a *_NEEDS_HOST_PYTHON variable that can be set either
to 'python2' or 'python3'. This variable can be set by any package
using the host-python-package infrastructure to force the python
interpreter for the build. This variable also takes care of setting
the right host-python dependency.
This *_NEEDS_HOST_PYTHON variable only affects packages using the
host-python-package infrastructure.
If some configure/build/install commands are overloaded in the *.mk
file, the right python interpreter should be explicitly called.
If the package defines some tool variable (eg.: SCONS), the variable
should explicitly call the right python interpreter.
[Thomas:
- fixes to the commit log and documentation suggested by Yann
- rename the variable from <pkg>_FORCE_HOST_PYTHON to
<pkg>_NEEDS_HOST_PYTHON, as suggested by Yann
- do not allow any other value than python2 and python3 in
<pkg>_NEEDS_HOST_PYTHON, as suggested by Yann.]
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
This commit improves the Python package infrastructure to allow Python
packages to be built with Python 3. The changes are fairly simple:
* Use either PYTHON_PATH or PYTHON3_PATH as the PYTHONPATH depending
on which Python is used.
* Depend on host-python or host-python3 and python or python3
depending on which Python is used.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Thanks to the bump of Python 2.x, distutilscross is no longer needed
to achieve cross-compilation for setuptools packages. The host Python
2.x interpreter can be tricked into using the target compiler thanks
to pointing it to a different sysconfigdata module, which is achieved
using PYTHON_PATH.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Thanks to the Python 2.x bump, it is no longer needed to pass
PYTHONCPREFIX, and CROSS_COMPILING when building third-party Python
modules.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
With the upcoming bump of Python 2.x, it will become important that
the PYTHONPATH is passed whenever we build third-party packages, be
they using the distutils build mechanism, or the setuptools build
mechanism. This is because passing PYTHONPATH is what will allow
Python to find a special Python module that contains all the
compiler/library/headers definitions that are relevant when
cross-compiling.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Now that the Python package exposes its PYTHON_PATH variable, we can
use it in the package infrastructure. This prepares both the upcoming
bump of Python 2.x, and the introduction of Python 3 support in the
Python package infrastructure.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This commit makes the dependency from the target toolchain explicit.
This way we can buid from command line a package that use
inner-generic-package right after the configuration phase, example:
make clean <package-name>
Also remove TARGETS_ALL because the only purpose was to add toolchain
dependency so it's superseded by this commit.
To prevent circular dependency add the new variable
<pkgname>_ADD_TOOLCHAIN_DEPENDENCY to avoid adding the toolchain
dependency for toolchain packages.
This is also a step forward supporting top-level parallel make.
Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The fourth parameter to inner-generic-package is no longer used. Removing
this parameters requires renaming all usages of $(5) to $(4), and updating
the calls to inner-generic-package (and equivalent for the other package
infrastructures).
Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When calling make 'functions', the $(call) keyword is only needed if the
function takes arguments. For pkgdir, pkgname and pkgparentdir this is not
the case, so we can remove the call to make things more readable.
Suggested-by: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This commit improves the cross-compilation patches we have on top of
Python, to fix the problem of host library paths leaking into the
build of target modules, as seen at:
http://autobuild.buildroot.org/results/fcc/fccd7e08cd9d4713eb4208097dd48c5ab25749bc/build-end.loghttp://autobuild.buildroot.org/results/0bd/0bda780bf4b759b12edec26ac20b88cde617db4d/build-end.log
To do so, it ensures that the right python2.7/config/Makefile is used
when building target modules, and adjusts at runtime the paths read
from this Makefile if we are cross-compiling.
In addition, it installs the pgen program into the host directory, and
points the target python build to use python and pgen from $(HOST_DIR)
instead of from the host python source directory, which looks cleaner.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Make the python packing a bit easier to use by providing a detailed error
message if <pkg>_SETUP_TYPE isn't set to a valid value.
At the same time adjust the error message used when <pkg>_SETUP_TIME isn't
set to list the exact variable name that should be set, and don't talk about
'Unknown' as it isn't printed when the variable isn't set (or is set to the
empty string).
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
[Peter: fix s/BUILD_TYPE/SETUP_TYPE/ typo in manual as noted by Samuel]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
