package/dmalloc: don't use SSP

dmalloc directly calls into $(LD) to generate a shared library our of
the static one.

To detect what command it should run, ./configure tries various
incantations of ld with various command line options until one does not
fail. One of those is (basically):
    $(LD) --whole-archive -o contest.o.t contest.a

This makes ./configure conclude what the command to link a shared
library in the Makefile should be, and thus stores that in a variable:
    shlinkargs='$(LD) --whole-archive -o $@'

... which is then AC_SUBST()ed into Makefile.in with a rule like:

    $(SHLIB): $(LIBRARY)
        @shlinkargs@ $(LIRARY)

which once substiuted, gives:

    $(SHLIB): $(LIBRARY)
        $(LD) --whole-archive -o $@ $(LIRARY)

However, when SSP is enabled, the __stack_chk_fail_local and co symbols
are provided by additional libraries or object files, and that is the
responsibility of gcc to pass those when linking. But as dmalloc
directly calls ld, it misses those.

Changing dmalloc to use $(CC) is not trivial, however.

First, we can't pass LD=$(TARGET_CC), otherwise the whole package
explodes [0]: indeed --whole-archive is unknown to gcc, so it must be
passed as -Wl,--whole archive instead. So we'd need to add a new test
that uses $(CC), like so:
    $(CC) -Wl,--whole-archive -o contest.o.t contest.a

However, in that case, gcc does pass additional libs/objs (like, for
eample, the SSP ones) to the linker. But then those are also included
in the whole-archive section. This causes the linker to add all the
symbols form those libs/objs, even those not needed for SSP; on some
archs, like PPC, that may require floating point symbols (__muldiv3 et
al.) which are in another library, and thus the linker can't find them.

The proper solution wouild be to add -Wl,--no-whole-archive, but that
would have to be added _after_ the library we want to link, i.e.e we
should be able to evntually run:

    $(CC) -Wl,--whole-archive -o $@ $(LIRARY) -Wl,--no-whole-archive

That would require that we introduce a new variable that is added
_after_ the $(LIBRARY), e.g. @shlinkargs_post@ or so...

This is a bigger endeavour than we want to pursue...

Since dmalloc is a debugging utility, it is not supposed to go into
production builds, and during debugging, it would not be surprising that
it needs to poke around arrays to debug them.

So, we go the easier route: disable SSP altogether.

[0] with lots of nice colors, but that's not the point, is it?

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
(cherry picked from commit 3481674ee3)
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>

package/dmalloc/dmalloc.mk

@@ -14,6 +14,13 @@ DMALLOC_LICENSE_FILES = LICENSE.txt
 DMALLOC_INSTALL_STAGING = YES
 DMALLOC_CFLAGS = $(TARGET_CFLAGS)
 
+# dmalloc uses $(LD) to link, and thus misses the object files or libs that
+# are needed to provide the __stack_chk_fail_local and co. symbols. Changing
+# to use $(CC) is really more complex that we'd like. Since dmalloc is
+# involved in debugging memory allocation, it is not expected to be a
+# production library, so we do not care that much that it has SSP.
+DMALLOC_CFLAGS += -fno-stack-protector
+
 ifeq ($(BR2_STATIC_LIBS),y)
 DMALLOC_CONF_OPTS += --disable-shlib
 else