This patch fixes a crash that happens when testing rfc4543(gcm(aes))
Unable to handle kernel paging request for data at address 0xf59b3420
Faulting instruction address: 0xc0012994
Oops: Kernel access of bad area, sig: 11 [#1]
BE PowerPC 44x Platform
Modules linked in: tcrypt(+) crypto4xx [...]
CPU: 0 PID: 0 Comm: swapper Tainted: G O 4.17.0-rc1+ #23
NIP: c0012994 LR: d3077934 CTR: 06026d49
REGS: cfff7e30 TRAP: 0300 Tainted: G O (4.17.0-rc1+)
MSR: 00029000 <CE,EE,ME> CR: 44744822 XER: 00000000
DEAR: f59b3420 ESR: 00000000
NIP [c0012994] __dma_sync+0x58/0x10c
LR [d3077934] crypto4xx_bh_tasklet_cb+0x188/0x3c8 [crypto4xx]
__dma_sync was fed the temporary _dst that crypto4xx_build_pd()
had in it's function stack. This clearly never worked.
This patch therefore overhauls the code from the original driver
and puts the temporary dst sg list into aead's request context.
Fixes: a0aae821ba ("crypto: crypto4xx - prepare for AEAD support")
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patch fixes cts(cbc(aes)) test when cbc-aes-ppc4xx is used.
alg: skcipher: Test 1 failed (invalid result) on encryption for cts(cbc-aes-ppc4xx)
00000000: 4b 10 75 fc 2f 14 1b 6a 27 35 37 33 d1 b7 70 05
00000010: 97
alg: skcipher: Failed to load transform for cts(cbc(aes)): -2
The CTS cipher mode expect the IV (req->iv) of skcipher_request
to contain the last ciphertext block after the {en,de}crypt
operation is complete.
Fix this issue for the AMCC Crypto4xx hardware engine.
The tcrypt test case for cts(cbc(aes)) is now correctly passed.
name : cts(cbc(aes))
driver : cts(cbc-aes-ppc4xx)
module : cts
priority : 300
refcnt : 1
selftest : passed
internal : no
type : skcipher
async : yes
blocksize : 16
min keysize : 16
max keysize : 32
ivsize : 16
chunksize : 16
walksize : 16
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patch adds support for the aes-ctr skcipher.
name : ctr(aes)
driver : ctr-aes-ppc4xx
module : crypto4xx
priority : 300
refcnt : 1
selftest : passed
internal : no
type : skcipher
async : yes
blocksize : 16
min keysize : 16
max keysize : 32
ivsize : 16
chunksize : 16
walksize : 16
The hardware uses only the last 32-bits as the counter while the
kernel tests (aes_ctr_enc_tv_template[4] for example) expect that
the whole IV is a counter. To make this work, the driver will
fallback if the counter is going to overlow.
The aead's crypto4xx_setup_fallback() function is renamed to
crypto4xx_aead_setup_fallback.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The ablkcipher APIs have been effectively deprecated since [1].
This patch converts the crypto4xx driver to the new skcipher APIs.
[1] <https://www.spinics.net/lists/linux-crypto/msg18133.html>
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patch provides a cheap 2MiB/s+ (~ 6%) performance
improvement over the current code. This is because the
compiler can now optimize several endian swap memcpy.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Use dma_zalloc_coherent for allocating zeroed
memory and remove unnecessary memset function.
Done using Coccinelle.
Generated-by: scripts/coccinelle/api/alloc/kzalloc-simple.cocci
0-day tested with no failures.
Signed-off-by: Himanshu Jha <himanshujha199640@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The ccm-aes-ppc4xx now fails one of testmgr's expected
failure test cases as such:
|decryption failed on test 10 for ccm-aes-ppc4xx:
|ret was 0, |expected -EBADMSG
It doesn't look like the hardware sets the authentication failure
flag. The original vendor source from which this was ported does
not have any special code or notes about why this would happen or
if there are any WAs.
Hence, this patch converts the aead_done callback handler to
perform the icv check in the driver. And this fixes the false
negative and the ccm-aes-ppc4xx passes the selftests once again.
|name : ccm(aes)
|driver : ccm-aes-ppc4xx
|module : crypto4xx
|priority : 300
|refcnt : 1
|selftest : passed
|internal : no
|type : aead
|async : yes
|blocksize : 1
|ivsize : 16
|maxauthsize : 16
|geniv : <none>
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
KBUILD_MODNAME provides the same value.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
crypto4xx_device's name variable is not set to anything.
The common devname for request_irq seems to be the module
name. This will fix the seemingly anonymous interrupt
entry in /proc/interrupts for crypto4xx.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patch adds support for the crypto4xx RevB cores
found in the 460EX, 460SX and later cores (like the APM821xx).
Without this patch, the crypto4xx driver will not be
able to process any offloaded requests and simply hang
indefinitely.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
It is possible to avoid the ce_base null pointer check in the
drivers' interrupt handler routine "crypto4xx_ce_interrupt_handler()"
by simply doing the iomap in front of the IRQ registration.
This way, the ce_base will always be valid in the handler and
a branch in an critical path can be avoided.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patch adds aes-gcm support to crypto4xx.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patch enhances existing interfaces and
functions to support AEAD ciphers in the next
patches.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Thanks to the big overhaul of crypto4xx_build_pd(), the request-local
sa_in, sa_out and state_record allocation can be simplified.
There's no need to setup any dma coherent memory anymore and
much of the support code can be removed.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
If the crypto4xx device is continuously loaded by dm-crypt
and ipsec work, it will start to work intermittent after a
few (between 20-30) seconds, hurting throughput and latency.
This patch contains various stability improvements in order
to fix this issue. So far, the hardware has survived more
than a day without suffering any stalls under the continuous
load.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
crypto4xx_core.c:179:6: warning: symbol 'crypto4xx_free_state_record'
was not declared. Should it be static?
crypto4xx_core.c:331:5: warning: symbol 'crypto4xx_get_n_gd'
was not declared. Should it be static?
crypto4xx_core.c:652:6: warning: symbol 'crypto4xx_return_pd'
was not declared. Should it be static?
crypto4xx_return_pd() is not used by anything. Therefore it is removed.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patch overhauls and fixes code related to crypto4xx_build_pd()
* crypto4xx_build_pd() did not handle chained source scatterlist.
This is fixed by replacing the buggy indexed-access of &src[idx]
with sg_next() in the gather array setup loop.
* The redundant is_hash, direction, save_iv and pd_ctl members
in the crypto4xx_ctx struct have been removed.
- is_hash can be derived from the crypto_async_request parameter.
- direction is already part of the security association's
bf.dir bitfield.
- save_iv is unused.
- pd_ctl always had the host_ready bit enabled anyway.
(the hash_final case is rather pointless, since the ahash
code has been deactivated).
* make crypto4xx_build_pd()'s caller responsible for converting
the IV to the LE32 format.
* change crypto4xx_ahash_update() and crypto4xx_ahash_digest() to
initialize a temporary destination scatterlist. This allows the
removal of an ugly cast of req->result (which is a pointer to an
u8-array) to a scatterlist pointer.
* change crypto4xx_build_pd() return type to int. After all
it returns -EINPROGRESS/-EBUSY.
* fix crypto4xx_build_pd() thread-unsafe sa handling.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The hardware expects that the keys, IVs (and inner/outer hashes)
are in the le32 format.
This patch changes all hardware interface declarations to use
the correct LE32 data format for each field.
In order to pass __CHECK_ENDIAN__ checks, crypto4xx_memcpy_le
has to be honest about the endianness of its parameters.
The function was split and moved to the common crypto4xx_core.h
header. This allows the compiler to generate better code if the
sizes/len is a constant (various *_IV_LEN).
Please note that the hardware isn't consistent with the endiannes
of the save_digest field in the state record struct though.
The hashes produced by GHASH and CBC (for CCM) will be in LE32.
Whereas md5 and sha{1/,256,...} do not need any conversion.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Previously, If the crypto4xx driver used all available
security contexts, it would simply refuse new requests
with -EAGAIN. CRYPTO_TFM_REQ_MAY_BACKLOG was ignored.
in case of dm-crypt.c's crypt_convert() function this was
causing the following errors to manifest, if the system was
pushed hard enough:
| EXT4-fs warning (dm-1): ext4_end_bio:314: I/O error -5 writing to ino ..
| EXT4-fs warning (dm-1): ext4_end_bio:314: I/O error -5 writing to ino ..
| EXT4-fs warning (dm-1): ext4_end_bio:314: I/O error -5 writing to ino ..
| JBD2: Detected IO errors while flushing file data on dm-1-8
| Aborting journal on device dm-1-8.
| EXT4-fs error : ext4_journal_check_start:56: Detected aborted journal
| EXT4-fs (dm-1): Remounting filesystem read-only
| EXT4-fs : ext4_writepages: jbd2_start: 2048 pages, inode 498...; err -30
(This did cause corruptions due to failed writes)
To fix this mess, the crypto4xx driver needs to notifiy the
user to slow down. This can be achieved by returning -EBUSY
on requests, once the crypto hardware was falling behind.
Note: -EBUSY has two different meanings. Setting the flag
CRYPTO_TFM_REQ_MAY_BACKLOG implies that the request was
successfully queued, by the crypto driver. To achieve this
requirement, the implementation introduces a threshold check and
adds logic to the completion routines in much the same way as
AMD's Cryptographic Coprocessor (CCP) driver do.
Note2: Tests showed that dm-crypt starved ipsec traffic.
Under load, ipsec links dropped to 0 Kbits/s. This is because
dm-crypt's callback would instantly queue the next request.
In order to not starve ipsec, the driver reserves a small
portion of the available crypto contexts for this purpose.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
I used aes-cbc as a template for ofb. But sadly I forgot
to update set_key method to crypto4xx_setkey_aes_ofb().
this was caught by the testmgr:
alg: skcipher: Test 1 failed (invalid result) on encr. for ofb-aes-ppc4xx
00000000: 76 49 ab ac 81 19 b2 46 ce e9 8e 9b 12 e9 19 7d
00000010: 50 86 cb 9b 50 72 19 ee 95 db 11 3a 91 76 78 b2
00000020: 73 be d6 b8 e3 c1 74 3b 71 16 e6 9e 22 22 95 16
00000030: 3f f1 ca a1 68 1f ac 09 12 0e ca 30 75 86 e1 a7
With the correct set_key method, the aes-ofb cipher passes the test.
name : ofb(aes)
driver : ofb-aes-ppc4xx
module : crypto4xx
priority : 300
refcnt : 1
selftest : passed
internal : no
type : ablkcipher
async : yes
blocksize : 16
min keysize : 16
max keysize : 32
ivsize : 16
geniv : <default>
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patch improves the readability of various functions,
by replacing various void* pointers declarations with
their respective structs *. This makes it possible to go
for the eye-friendly array-indexing methods.
Signed-off-by: Christian Lamparter <chunkeey@googlemail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
If one of the later memory allocations in rypto4xx_build_pdr()
fails: dev->pdr (and/or) dev->pdr_uinfo wouldn't be freed.
crypto4xx_build_sdr() has the same issue with dev->sdr.
Signed-off-by: Christian Lamparter <chunkeey@googlemail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
scatter_buffer_size is always set to PPC4XX_SD_BUFFER_SIZE.
I don't think there's any point in keeping the variable
around.
Signed-off-by: Christian Lamparter <chunkeey@googlemail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patch refactors the crypto4xx_copy_pkt_to_dst() to use
scatterwalk_map_and_copy() to copy the processed data between
the crypto engine's scatter ring buffer and the destination
specified by the ablkcipher_request.
This also makes the crypto4xx_fill_one_page() function redundant.
Signed-off-by: Christian Lamparter <chunkeey@googlemail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The crypto engine supports more than just aes-cbc. This patch
enables the remaining AES block cipher modes that pass the
testmanager's test vectors.
Signed-off-by: Christian Lamparter <chunkeey@googlemail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
crypto4xx_put_pd_to_pdr() already clears the flag.
Signed-off-by: Christian Lamparter <chunkeey@googlemail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The security offload function is performed by a cryptographic
engine core attached to the 128-bit PLB (processor local bus)
with builtin DMA and interrupt controllers. This, I think,
satisfies the requirement for the CRYPTO_ALG_KERN_DRIVER_ONLY
flag.
Signed-off-by: Christian Lamparter <chunkeey@googlemail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patch removes several unused code and definitons
(structs, variables, ...).
Signed-off-by: Christian Lamparter <chunkeey@googlemail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
alg entries are only added to the list, after the registration
was successful. If the registration failed, it was never added
to the list in the first place.
Signed-off-by: Christian Lamparter <chunkeey@googlemail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
If 'kzalloc' fails, we return 0 which means success.
return -ENOMEM instead as already done a few lines above.
Signed-off-by: Christophe JAILLET <christophe.jaillet@wanadoo.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
trivial spelling mistake, missing r, rename to ce_ring_control
Signed-off-by: Colin Ian King <colin.king@canonical.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The size used in 'dma_free_coherent()' looks un-initialized here.
ctx->sa_len is set a few lines below and is apparently not set by the
caller.
So use 'size' as in the corresponding 'dma_alloc_coherent()' a few lines
above.
This has been spotted with coccinelle, using the following script:
////////////////////
@r@
expression x0, x1, y0, y1, z0, z1, t0, t1, ret;
@@
* ret = dma_alloc_coherent(x0, y0, z0, t0);
...
* dma_free_coherent(x1, y1, ret, t1);
@script:python@
y0 << r.y0;
y1 << r.y1;
@@
if y1.find(y0) == -1:
print "WARNING: sizes look different: '%s' vs '%s'" % (y0, y1)
////////////////////
Signed-off-by: Christophe JAILLET <christophe.jaillet@wanadoo.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patch integrates the ppc4xx-rng driver into the existing
crypto4xx. This is because the true random number generator
is controlled and part of the security core.
Signed-off-by: Christian Lamparter <chunkeey@googlemail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The sg_nents_for_len() function could fail, this patch add a check for
its return value.
Signed-off-by: LABBE Corentin <clabbe.montjoie@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The get_sg_count function of amcc is the same as sg_nents_for_len from
lib/scatterlist.c
Signed-off-by: LABBE Corentin <clabbe.montjoie@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This platform driver has a OF device ID table but the OF module
alias information is not created so module autoloading won't work.
Signed-off-by: Luis de Bethencourt <luis@debethencourt.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Compare pointer-typed values to NULL rather than 0.
The semantic patch that makes this change is available
in scripts/coccinelle/null/badzero.cocci
Signed-off-by: Fabio Estevam <fabio.estevam@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Even if bus is not hot-pluggable, the devices can be bound and unbound
from the driver via sysfs, so we should not be using __init/__exit
annotations on probe() and remove() methods. The only exception is
drivers registered with platform_driver_probe() which specifically
disables sysfs bind/unbind attributes.
Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
References to __exit functions must be wrapped with __exit_p.
Signed-off-by: Jean Delvare <jdelvare@suse.de>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Robert Jennings <rcj@linux.vnet.ibm.com>
Cc: Marcelo Henrique Cerri <mhcerri@linux.vnet.ibm.com>
Cc: Fionnuala Gunter <fin@linux.vnet.ibm.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Driver core sets it to NULL upon probe failure or release.
Signed-off-by: Sachin Kamat <sachin.kamat@linaro.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Commit b5b4bb3f6a (of: only include prom.h on sparc) removed implicit
includes of of_*.h headers by powerpc's prom.h. Some components were
missed in initial clean-up patch, so add the necessary includes to fix
powerpc builds.
Signed-off-by: Rob Herring <rob.herring@calxeda.com>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Matt Mackall <mpm@selenic.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Vinod Koul <vinod.koul@intel.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-ide@vger.kernel.org
Cc: linux-crypto@vger.kernel.org
Convert a nonnegative error return code to a negative one, as returned
elsewhere in the function.
A simplified version of the semantic match that finds this problem is as
follows: (http://coccinelle.lip6.fr/)
// <smpl>
(
if@p1 (\(ret < 0\|ret != 0\))
{ ... return ret; }
|
ret@p1 = 0
)
... when != ret = e1
when != &ret
*if(...)
{
... when != ret = e2
when forall
return ret;
}
// </smpl>
Signed-off-by: Peter Senna Tschudin <peter.senna@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Move the err_request_irq error label up to reflect that tasklet_init and
irq_of_parse_and_map have taken place.
Signed-off-by: Julia Lawall <Julia.Lawall@lip6.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patch converts the drivers in drivers/crypto/* to use the
module_platform_driver() macro which makes the code smaller and a bit
simpler.
Cc: James Hsiao <jhsiao@amcc.com>
Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Cc: Kim Phillips <kim.phillips@freescale.com>
Cc: "David S. Miller" <davem@davemloft.net>
Signed-off-by: Axel Lin <axel.lin@gmail.com>
Acked-by: Vladimir Zapolskiy <vzapolskiy@gmail.com>
Acked-by: Jamie Iles <jamie@jamieiles.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The Security function on the AMCC SoCs has multiple engines within a
single MMIO range. The crypto driver currently enables the 3DES
functionality by doing a blind write to the device control register.
This can unintentionally disable other functions like the PKA or TRNG
when the driver is loaded.
Perform a read/modify/write to enable the 3DES function instead.
Signed-off-by: Josh Boyer <jwboyer@linux.vnet.ibm.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Get rid of old users of of_platform_driver in arch/powerpc. Most
of_platform_driver users can be converted to use the platform_bus
directly.
Signed-off-by: Grant Likely <grant.likely@secretlab.ca>
of_device is just an alias for platform_device, so remove it entirely. Also
replace to_of_device() with to_platform_device() and update comment blocks.
This patch was initially generated from the following semantic patch, and then
edited by hand to pick up the bits that coccinelle didn't catch.
@@
@@
-struct of_device
+struct platform_device
Signed-off-by: Grant Likely <grant.likely@secretlab.ca>
Reviewed-by: David S. Miller <davem@davemloft.net>
Kees Cook