This commit modifies the toolchain-wrapper to make sure that the
CCACHE_COMPILERCHECK and CCACHE_BASEDIR are only set if ccache support
is enabled. Indeed, when BR2_USE_CCACHE is not set (or to a value
different than 1), we don't call the compiler with ccache, so there is
no reason to set those ccache environment variables, and they could
potentially conflict with a separate usage of ccache, outside of
Buildroot, for example when using the Buildroot SDK.
In particular, the value of CCACHE_BASEDIR doesn't not make any sense
when the Buildroot toolchain is not used during the Buildroot build,
as it points to the output directory $(BASE_DIR).
We pay attention to also not show those variables as being set in the
BR2_DEBUG_WRAPPER dump.
To help a little bit with this, a ccache_enabled boolean is introduced
to indicate when ccache is being used.
There is still quite a bit of #ifdef-ery involved, but it's not easy
to find a simpler way to organize the code.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-By: Vincent Fazio <vfazio@xes-inc.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
When we introduced support for the paranoid check of unsafe libraries
and headers path with commit 4ac8f78d37 (Add option for paranoid
unsafe path checking) back in 2014, we made it optional, as we expected
that would break quite a few packages.
Now, almost 8 years later, we only have three packages that explicitly
reference the option (dillo, gnuradio, and libtalloc), either in a patch
or in their .mk.
The option has been enabled by default since 2016, with 61c8854cef
(toolchain: enable paranoid unsafe path check by default), and that has
not triggered many build failures in a while.
The minimal defconfig used by test-pkg has also had it enabled as of
b6c98b3549 (minimal.config: add BR2_COMPILER_PARANOID_UNSAFE_PATH=y)
in 2017.
It is time to make that globally unconditional now.
There is still a remnant, in our binutils patches. As our toolchain may
get used outside of Buildroot, people may got the expectation that path
poisoning is only a warning, so we keep the current behaviour.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Romain Naour <romain.naour@gmail.com>
Acked-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Until now, when BR2_CCACHE=y, ccache support was built into the
toolchain wrapper, and used regardless of whether the toolchain is
using during the Buildroot build itself, or later as part of the SDK.
However, having ccache support forcefully enabled in the SDK can
really be surprising, and is certainly unexpected for a
cross-compilation toolchain. This can be particularly surprising as
the ccache cache directory may be hardcoded in the ccache binary to
point to a folder that does not make sense on the SDK user's machine.
So what this commit does is create a BR2_USE_CCACHE variable, which
when set to 1 tells the toolchain wrapper to use ccache. Not defining
the variable, or specifying any other value that 1 causes the
toolchain wrapper to not use ccache. The main Buildroot Makefile is
modified to export BR2_USE_CCACHE = 1 when ccache support is enabled,
so that ccache is used during the Buildroot build.
However, when someone will use the SDK outside of Buildroot, the
toolchain wrapper will not use ccache.
The BR2_USE_CCACHE variable is only conditionally enabled in the main
Makefile (via ?=) so that it can be overridden in the environment if
one wants to quickly test disabling ccache in a ccache-enabled
Buildroot configuration. This is the scenario that was considered in
commit 792f1278e3 ("toolchain-wrapper:
support change of BR2_CCACHE"), which added the BR_NO_CCACHE variable.
The BR_NO_CCACHE variable is no longer needed, and replaced by this
BR2_USE_CCACHE variable.
Signed-off-by: Paul Cercueil <paul@crapouillou.net>
[Thomas: almost entirely rework the implementation and commit log]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When using precompiled headers, changing any macros defined on the
command line will invalidate the precompiled header. With
toolchain-wrapper adding __DATE__ and __TIME__, any commits to Buildroot
will invalidate incremental builds regardless of whether the precompiled
header actually uses those values (affecting _OVERRIDE_SRCDIR).
GCC-7 and later support SOURCE_DATE_EPOCH and use it to define __DATE__
and __TIME__ internally, avoiding any impact on precompiled headers.
Disable the custom handling in toolchain-wrapper if GCC is version 7 or
newer.
Signed-off-by: John Keeping <john@metanate.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
On some legacy systems, the X11 headers and libs are in /usr/X11R66/include
and /usr/X11R66/lib, and of course, some packages are trying to be smart
and use those paths (even when they do not exist).
Add those to the list of unsafe paths to check in the toolchain wrapper.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Most, but not all our C code follows the Linux kernel code style (as
documented in Documentation/process/coding-style.rst). Adjust the few
places doing differently:
- Braces:
..but the preferred way, as shown to us by the prophets Kernighan
and Ritchie, is to put the opening brace last on the line
- Spaces after keywords:
Use a space after (most) keywords
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Reviewed-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Tested-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
In commit 7484c1c3b8 (toolchain/toolchain-wrapper: add BR2_RELRO_),
we added the PIC/PIE flags, but based on the RELRO_FULL condition.
It is however totally possible to do a PIC/PIE executable without
RELRO_FULL, as it is also valid to do a PIC/PIE build with RELRO_PARTIAL.
Add a new option that now governs the PIC/PIE flags.
Note: it is unknown if RELRO_FULL really needs PIC/PIE or not, so we
keep the current situation, where RELRO-FULL forces PIC/PIE compilation.
Decoupling can come later from an interested party.
Signed-off-by: "Yann E. MORIN" <yann.morin@orange.com>
Cc: Matt Weber <matthew.weber@rockwellcollins.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Reviewed-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Currently, we repeat all the SSP level selection deep down to the
toolchain wrapper itself, where we eventually translate it to the
actual SSP option to use. This is a bit redundant.
Additionally, we will want to check that the toolchain actually
supports that option (for those toolchain where it was backported).
So, move the translation into kconfig, and add the qstrip'ed value
to the additional flags passed to the wrapper. Add it before
user-supplied opitons, to keep the previous behaviour (and allow
anyone crazy-enough to override it with BR2_TARGET_OPTIMIZATION).
Signed-off-by: "Yann E. MORIN" <yann.morin@orange.com>
Cc: Matt Weber <matthew.weber@rockwellcollins.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Reviewed-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Migrate the stack protection flag management into the wrapper.
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The RELRO/PIE flags are currently passed via CFLAGS/LDFLAGS and this patch
proposes moving them to the toolchain wrapper.
(1) The flags should _always_ be passed, without leaving the possibility
for any package to ignore them. I.e, when BR2_RELRO_FULL=y is used
in a build, all executables should be built PIE. Passing those
options through the wrapper ensures they are used during the build
of all packages.
(2) Some options are incompatible with -fPIE. For example, when
building object files for a shared libraries, -fPIC is used, and
-fPIE shouldn't be used in combination with -fPIE. Similarly, -r
or -static are directly incompatible as they are different link
time behaviors then the intent of PIE. Passing those options
through the wrapper allows to add some "smart" logic to only pass
-fPIE/-pie when relevant.
(3) Some toolchain, kernel and bootloader packages may want to
explicitly disable PIE in a build where the rest of the userspace
has intentionally enabled it. The wrapper provides an option
to key on the -fno-pie/-no-pie and bypass the appending of RELRO
flags.
The current Kernel and U-boot source trees include this option.
8438ee76b06ace36e19a
If using PIE with a older Kernel and/or U-boot version, a backport of these
changes might be required. However this patchset also uses the
__KERNEL__ and __UBOOT__ defines as a way to disable PIE.
NOTE: The current implementation via CFLAGS/LDFLAGS has caused some
build time failures as the conditional logic doesn't yet exist in
Buildroot:
https://bugs.busybox.net/show_bug.cgi?id=11206https://bugs.busybox.net/show_bug.cgi?id=11321
Good summary of the most common build failures related to
enabling pie: https://wiki.ubuntu.com/SecurityTeam/PIE
[Peter: minor cleanups]
Signed-off-by: Matthew Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Since gcc 4.6, GCC deprecated -mfused-madd, -ffp-contract=off should
be used for the Xburst workaround.
Tested with the MIPS Sourcery 2011.03 toolchain (based on gcc 4.5),
the toolchain wrapper uses -mno-fused-madd, as expected:
$ BR2_DEBUG_WRAPPER=2 ./output/host/bin/mips-linux-gnu-gcc -o toto toto.c
Toolchain wrapper executing:
'/home/thomas/toolchains/mips-2011.03/bin/mips-linux-gnu-gcc'
'--sysroot'
'/home/thomas/projets/buildroot/output/host/mipsel-buildroot-linux-gnu/sysroot'
'-mabi=32'
'-msoft-float'
'-mno-fused-madd'
'-EL'
'-march=mips32r2'
'-o'
'toto'
'toto.c'
And with the MIPS Sourcery 2012.09 toolchain (based on gcc 4.7), the
toolchain wrapper uses -ffp-contract=off, as expected:
$ BR2_DEBUG_WRAPPER=2 ./output/host/bin/mips-linux-gnu-gcc -o toto toto.c
Toolchain wrapper executing:
'/home/thomas/toolchains/mips-2012.09/bin/mips-linux-gnu-gcc'
'--sysroot'
'/home/thomas/projets/buildroot/output/host/mipsel-buildroot-linux-gnu/sysroot'
'-mabi=32'
'-msoft-float'
'-ffp-contract=off'
'-EL'
'-march=mips32r2'
'-o'
'toto'
'toto.c'
Fixes the ci20_defconfig build:
https://gitlab.com/buildroot.org/buildroot/-/jobs/60303132
Signed-off-by: Waldemar Brodkorb <wbx@openadk.org>
[Thomas: rework to continue supporting pre-gcc-4.6 toolchains, extend
the commit log after doing more testing.]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Ezequiel Garcia <ezequiel@vanguardiasur.com.ar>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Starting with version 7, gcc automatically recognises and enforces the
environment variable SOURCE_DATE_EPOCH, and fakes __DATE__ and __TIME__
accordingly, to produce reproducible builds (at least in regards to date
and time).
However, older gcc versions do not offer this feature.
So, we use our toolchain wrapper to force-feed __DATE__ and __TIME__ as
macros, which will take precedence over those that gcc may compute
itself. We compute them according to the specs:
https://reproducible-builds.org/specs/source-date-epoch/https://gcc.gnu.org/onlinedocs/cpp/Standard-Predefined-Macros.html
Since we define macros otherwise internal to gcc, we have to tell it not
to warn about that. The -Wno-builtin-macro-redefined flag was introduced
in gcc-4.4.0. Therefore, we make BR2_REPRODUCIBLE depend on GCC >= 4.4.
gcc-7 will ignore SOURCE_DATE_EPOCH when __DATE__ and __TIME__ are
user-defined. Anyway, this is of no consequence: whether __DATE__ and
__TIME__ or SOURCE_DATE_EPOCH takes precedence, it would yield the
exact same end result since we use the same logic to compute it. Note
that we didn't copy the code for it from gcc so using the same logic
doesn't imply that we're inheriting GPL-3.0.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Jérôme Pouiller <jezz@sysmic.org>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
[Arnout: rewrite commit message]
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
MIPS32 support different FP modes (32,xx,64), so give the user the
opportunity to choose between them. That will cause host-gcc to be built
using the --with-fp-32=[32|xx|64] configure option. Also the
-mfp[32|xx|64] gcc option will be added to TARGET_CFLAGS and to the
toolchain wrapper.
FP mode option shouldn't be used for soft-float, so we add logic in the
toolchain wrapper if -msoft-float is among the arguments in order to not
append the -fp[[32|xx|64] option, otherwise the compilation may fail.
Information about FP modes here:
- https://sourceware.org/binutils/docs/as/MIPS-Options.html
- https://dmz-portal.imgtec.com/wiki/MIPS_O32_ABI_-_FR0_and_FR1_Interlinking#5._Generating_modeless_code
Signed-off-by: Vicente Olivert Riera <Vincent.Riera@imgtec.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
MIPS supports two different NaN encodings, legacy and 2008. Information
about MIPS NaN encodings can be found here:
https://sourceware.org/binutils/docs/as/MIPS-NaN-Encodings.html
NaN legacy is the only option available for R2 cores and older.
NaN 2008 is the only option available for R6 cores.
R5 cores can have either NaN legacy or NaN 2008, depending on the
implementation. So, if the user selects a generic R5 target architecture
variant, we show a choice menu with both options available. For well
known R5 cores we directly select the NaN enconding they use.
Signed-off-by: Vicente Olivert Riera <Vincent.Riera@imgtec.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Commit 14151d77af that eliminated
$(HOST_DIR)/usr seriously missed the toolchain-wrapper - only a single
reference was updated, the other three were missed. Commit
015d68c84c removed one more. This commit
finally removes the two remaining ones.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
The toolchain wrapper, when called through PATH, strips the last three
levels of /proc/self/exe to find HOST_DIR. However, after the host/usr
removal, this should be just two levels.
The toolchain wrapper has different logic for when it is called with a
full path (i.e. $HOST_DIR/usr/bin/arm-linux-gcc) then when it is called
through the PATH (i.e. just arm-linux-gcc). The latter is never used
internally in Buildroot, that's why this wasn't discovered through
testing.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Mark Jackson <mpfj-list@newflow.co.uk>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
We currently use $(HOST_DIR)/usr as the prefix for host packages. That
has a few disadvantages:
- There are some things installed in $(HOST_DIR)/etc and
$(HOST_DIR)/sbin, which is inconsistent.
- To pack a buildroot-built toolchain into a tarball for use as an
external toolchain, you have to pack output/host/usr instead of the
more obvious output/host.
- Because of the above, the internal toolchain wrapper breaks which
forces us to work around it (call the actual toolchain executable
directly). This is OK for us, but when used in another build system,
that's a problem.
- Paths are four characters longer.
To allow us to gradually eliminate $(HOST_DIR)/usr while building
packages, replace it with a symlink to .
The symlinks from $(HOST_DIR)/usr/$(GNU_TARGET_NAME) and
$(HOST_DIR)/usr/lib that were added previously are removed again.
Note that the symlink creation will break when $(HOST_DIR)/usr
already exists as a directory, i.e. when rebuilding in an existing
output directory. This is necessary: if we don't break it now, the
following commits (which remove the usr part from various variables)
_will_ break it.
At the same time as creating this symlink, we have to update the
external toolchain wrapper and the external toolchain symlinks to go
one directory less up. Indeed, $(HOST_DIR) is one level less up than
it was before.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
This reverts commit 76838f6341.
The commit referenced above explicitly states that the function was
copied as-is from the gcc source code at the time. And indeed that is
exactly how the function appeared in gcc in commit
e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934.
However, our toolchain wrapper is "GPLv2 only", while the file this function
was copied from is "GPLv3 or later". As such we can't include that function
and still comply to both licenses.
Furthermore, the code is far from optimum.
Since this feature is not release-critical, revert it until we re-implement
it.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Jérôme Pouiller <jezz@sysmic.org>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The use of the __DATE__ and __TIME__ macros are one of the most common
sources of non-reproducible binaries. In order to fix that, gcc 7 supports
the SOURCE_DATE_EPOCH variable:
https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934
This patch take advantage of toolchain-wrapper to provide support of
SOURCE_DATE_EPOCH to older gcc versions.
Function get_source_date_epoch() come directly from gcc git.
This work was sponsored by `BA Robotic Systems'.
[Peter: use sizeof for character array sizes, make function static,
extend commit message, add upstream gcc commit]
Signed-off-by: Jérôme Pouiller <jezz@sysmic.org>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
In 61cb120 (toolchain/wrapper: extend paranoid check to -isystem), we
introduced a {str,len} tuple to check the various arguments pased to
gcc, to avoid hard-coding an ever-growing, long list of those args
directly in the condition check.
Now, we're left with a long list of unsafe paths, somehow hidden within
the code, which can use the same mechanism we use for arguments.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
In 61cb120 (toolchain/wrapper: extend paranoid check to -isystem), we
introduced a {str,len} tuple to check the various arguments passed to
gcc, to avoid hard-coding an ever-growing, long list of those args
directly in the condition check.
It was made specific to the arguments (the structure member is named
'arg'), but can also be used to store the unsafe paths as well.
Also, that piece is almost un-documented.
Rename the structure member so that it is more generic, and add a bit of
comments to explain the whole of it.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
The Ingenic XBurst is a MIPS32R2 microprocessor.
It has a bug in the FPU that can generate incorrect results in certain
cases. The problem shows up when you have several fused madd
instructions in sequence with dependant operands.
Using the -mno-fused-madd option prevents gcc from emitting these
instructions. This patch adds changes to the toolchain wrapper to use
that option.
Signed-off-by: Vicente Olivert Riera <Vincent.Riera@imgtec.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Some packages, like libbsd, use -isystem flags to provide so-called
overrides to the system include files. In this particular case, this
is used in a .pc file, then used by antoher package; pkgconf does not
mangle this path; and eventually that other package ends up using
/usr/include/bsd to search for headers.
Our current toolchain wrapper is limited to looking for -I and -L, so
the paranoid check does not kick in.
Furthermore, as noticed by Arnout, there might be a bunch of other
so-unsafe options: -isysroot, -imultilib, -iquote, -idirafter, -iprefix,
-iwithprefix, -iwithprefixbefore; even -B and --sysroot are unsafe.
Extend the paranoid check to be able to check any arbitrary number of
potentially unsafe options:
- add a list of options to check for, each with their length,
- iterate over this list until we find a matching unsafe option.
Compared to previously, the list of options include -I and -L (which we
already had) extended with -idirafter, -iquote and -isystem, but leaving
all the others noticed by Arnout away, until we have a reason for
handling them.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Current, we only display the path that causes the paranoid failure. This
is sufficient, as we can fail only for -I and -L options, and it is thus
easy to infer from the path, which option is the culprit.
However, we're soon to add a new test for the -isystem option, and then
when a failure occurs, we would not know whether it was because of -I or
-isystem. Being able to differentiate both can be hugely useful to
track down the root cause for the unsafe path.
Add two new arguments to the check_unsafe_path() function: one with the
current-or-previous argument, one to specify whether it has the path in
it or not. Print that in the error message, instead of just the path.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
The Intel X1000 is the Pentium class microprocessor that ships with
Galileo Gen 1/2. This patch adds changes to arch and toolchain-wrapper
to omit the lock prefix for the X1000.
[Thomas: tweak commit log and Config.in help text.]
Signed-off-by: Ray Kinsella <ray.kinsella@intel.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
The buildroot internal toolchain now adds a wrapper. When we use a
buildroot toolchain as an external toolchain, we want to bypass this
wrapper and call the compiler directly, for two reasons:
1. The options added by the wrapper are not necessarily appropriate
when it is reused as an external toolchain. For instance, ccache
may have been enabled while building the toolchain but not when
using it as an external toolchain.
2. Currently, the wrapper expects to reside in .../usr/bin, but when
used as an external toolchain it will be in .../ext-toolchain/bin.
Therefore, the wrapper can't find the real binary and sysroot
anymore.
To bypass the wrapper, we check for the existence of *.br_real files in
the external toolchain directory. If any such file exists, the wrapper
will add the .br_real suffix for all the wrapped files. Note that the
wrapper doesn't check if the *.br_real exists for each individual
wrapped file, it just assumes that all wrapped files have a
corresponding .br_real. This is currently true but that may change in
the future of course.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Fixes#8386
We should check if BR_CROSS_PATH_ABS is defined, not if it evalutates to
true for the pre processor.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When building in a different output directory than the original build,
there will currently be a lot of ccache misses because in many cases
there is some -I/... absolute path in the compilation. Ccache has an
option CCACHE_BASEDIR to substitute absolute paths with relative paths,
so they wil be the same in the hash (and in the output).
Since there are some disadvantages to this path rewriting, it is made
optional as BR2_CCACHE_USE_BASEDIR. It defaults to y because the
usefulness of ccache is severely reduced without this option.
In addition to CCACHE_BASEDIR, we also substitute away the occurences
of $(HOST_DIR) in the calculation of the compiler hash. This is done
regardless of the setting of BR2_CCACHE_USE_BASEDIR because it's
quite harmless.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Our current ccache disables hashing of the compiler executable itself,
because using the default 'mtime' doesn't work in buildroot: we always
rebuild the compiler, so the mtime is always different, so the cache
always misses.
However, in the current situation, if a user changes the compiler
configuration (which would result in the compiler generating different
object files than before) and does 'make clean all', ccache may in fact
reuse object files from the previous run. This rarely gives problems,
because
(1) the cache expires quite quickly (it's only 1GB by default),
(2) radically changing compiler options will cause cache misses because
different header files are used,
(3) many compiler changes (e.g. changing -mtune) have little practical
effect because the resulting code is usually still compatible,
(4) we currently don't use CCACHE_BASEDIR, and almost all object files
will contain an absolute path (e.g. in debug info), so when
building in a different directory, most of it will miss,
(5) we do mostly build test, and many of the potential problems only
appear at runtime.
Still, when ccache _does_ use the wrong cached object files, the
effects are really weird and hard to debug. Also, we want reproducible
builds and obviously the above makes builds non-reproducible. So we
have a FAQ entry that warns against using ccache and tells the user to
clear the cache in case of problems.
Now that ccache is called from the toolchain wrapper, it is in fact
possible to at least use the 'mtime' compiler hash for the external
toolchain and for the host-gcc. Indeed, in this case, the compiler
executable comes from a tarball so the mtime will be a good reference
for its state. Therefore, the patch (sed script) that changes the
default from 'mtime' to 'none' is removed.
For the internal toolchain, we can do better by providing a hash of
the relevant toolchain options. We are only interested in things that
affect the compiler itself, because ccache also processes the header
files and it doesn't look at libraries because it doesn't cache the
link step, just compilation. Everything that affects the compiler
itself can nicely be summarised in $(HOST_GCC_FINAL_CONF_OPTS). Of
course, also the compiler source itself is relevant, so the source
tarball and all the patches are included in the hash. For this purpose,
a new HOST_GCC_XTENSA_OVERLAY_TAR is introduced.
The following procedure tests the ccache behaviour:
Use this defconfig:
BR2_arm=y
BR2_CCACHE=y
make
readelf -A output/build/uclibc-1.0.6/libc/signal/signal.os
-> Tag_CPU_name: "ARM926EJ-S"
Now make menuconfig, change variant into BR2_cortex_a9
make clean; make
readelf -A output/build/uclibc-1.0.6/libc/signal/signal.os
-> Tag_CPU_name: "ARM926EJ-S"
should be "Cortex-A9"
After this commit, it is "Cortex-A9".
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Danomi Manchego <danomimanchego123@gmail.com>
Cc: Károly Kasza <kaszak@gmail.com>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Romain Naour <romain.naour@openwide.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
By moving the ccache call to the toolchain wrapper, the following
scenario no longer works:
make foo-dirclean all BR2_CCACHE=
That's a sometimes useful call to check if some failure is perhaps
caused by ccache.
We can enable this scenario again by exporting BR_NO_CCACHE when
BR2_CCACHE is not set, and by handling this in the toolchain wrapper.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Tested-by: Romain Naour <romain.naour@openwide.fr>
Reviewed-by: Romain Naour <romain.naour@openwide.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Since we always have a toolchain wrapper now, we can move the ccache
call to the toolchain wrapper.
The hostcc ccache handling obviously stays.
The global addition of ccache to TARGET_CC/CXX is removed, but many
individual packages and infras still add it. This means we have a
chain like this: ccache -> toolchain-wrapper -> ccache -> gcc
However, this is fairly harmless: for cache misses, the inner ccache
just adds overhead and for cache hits, the inner ccache is never
called. Later patches will remove these redundant ccache calls.
As a side effect, perl now supports ccache as well.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Danomi Manchego <danomimanchego123@gmail.com>
Cc: Károly Kasza <kaszak@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
We have a toolchain wrapper for external toolchain, but it is also
beneficial for internal toolchains, for the following reasons:
1. It can make sure that BR2_TARGET_OPTIMIZATION is passed to the
compiler even if a package's build system doesn't honor CFLAGS.
2. It allows us to do the unsafe path check (i.e. -I/usr/include)
without patching gcc.
3. It makes it simpler to implement building each package with a
separate staging directory (per-package staging).
4. It makes it simpler to implement a compiler hash check for ccache.
The wrapper is reused from the external toolchain. A third CROSS_PATH_
option is added to the wrapper: in this case, the real executable is in
the same directory, with the extension .real.
The creation of the simple symlinks is merged with the creation of the
wrapper symlinks, otherwise part of the -gcc-ar handling logic would
have to be repeated.
The complex case-condition could be refactored with the one for the
external toolchain, but then it becomes even more complex because
they each have special corner cases. For example, the internal
toolchain has to handle *.real to avoid creating an extra indirection
after host-gcc-{final,initial}-rebuild.
Instead of creating the .real files, it would also have been possible
to install the internal toolchain in $(HOST_DIR)/opt, similar to what
we do for the external toolchain. However, then we would also have to
copy things to the sysroot and do more of the magic that the external
toolchain is doing. So keeping it in $(HOST_DIR)/usr/bin is much
simpler.
Note that gcc-initial has to be wrapped as well, because it is used for
building libc and we want to apply the same magic when building libc.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Fabio Porcedda <fabio.porcedda@gmail.com>
Cc: Jérôme Oufella <jerome.oufella@savoirfairelinux.com>
Reviewed-by: Romain Naour <romain.naour@openwide.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The toolchain wrapper will be reused for the internal toolchain, so it
belongs in the toolchain directory. Also, the ext- prefix is removed
from it. The build commands are moved to a new toolchain-wrapper.mk.
The wrapper arguments that are also relevant for the internal toolchain
wrapper are moved to toolchain-wrapper.mk, the rest stays in
toolchain-external.mk.
While we're at it, move the building of the toolchain wrapper to the
build step of toolchain-external. There is no specific reason to do
this, other than that it fits better semantically. Also remove the
MESSAGE call, otherwise we'd see:
>>> toolchain-external undefined Building
>>> toolchain-external undefined Building toolchain wrapper
/usr/bin/gcc ...
Having an extra "Building toolchain wrapper' message is pointless.
The useless condition on $(BR2_TARGET_OPTIMIZATION) is removed. It was
always true because it wasn't qstrip'ped first, so clearly it works
without that condition as well.
Also rewrapped some comments and removed the 'external' reference.
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Fabio Porcedda <fabio.porcedda@gmail.com>
Cc: Jérôme Oufella <jerome.oufella@savoirfairelinux.com>
Reviewed-by: Romain Naour <romain.naour@openwide.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Thomas Petazzoni