redonkable
/
alistair23-linux
1
0
Fork 0
Code Releases Activity

Merge branch 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6 

Pull crypto update from Herbert Xu:
 "API:
   - Add helper for simple skcipher modes.
   - Add helper to register multiple templates.
   - Set CRYPTO_TFM_NEED_KEY when setkey fails.
   - Require neither or both of export/import in shash.
   - AEAD decryption test vectors are now generated from encryption
     ones.
   - New option CONFIG_CRYPTO_MANAGER_EXTRA_TESTS that includes random
     fuzzing.

  Algorithms:
   - Conversions to skcipher and helper for many templates.
   - Add more test vectors for nhpoly1305 and adiantum.

  Drivers:
   - Add crypto4xx prng support.
   - Add xcbc/cmac/ecb support in caam.
   - Add AES support for Exynos5433 in s5p.
   - Remove sha384/sha512 from artpec7 as hardware cannot do partial
     hash"

[ There is a merge of the Freescale SoC tree in order to pull in changes
  required by patches to the caam/qi2 driver. ]

* 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6: (174 commits)
  crypto: s5p - add AES support for Exynos5433
  dt-bindings: crypto: document Exynos5433 SlimSSS
  crypto: crypto4xx - add missing of_node_put after of_device_is_available
  crypto: cavium/zip - fix collision with generic cra_driver_name
  crypto: af_alg - use struct_size() in sock_kfree_s()
  crypto: caam - remove redundant likely/unlikely annotation
  crypto: s5p - update iv after AES-CBC op end
  crypto: x86/poly1305 - Clear key material from stack in SSE2 variant
  crypto: caam - generate hash keys in-place
  crypto: caam - fix DMA mapping xcbc key twice
  crypto: caam - fix hash context DMA unmap size
  hwrng: bcm2835 - fix probe as platform device
  crypto: s5p-sss - Use AES_BLOCK_SIZE define instead of number
  crypto: stm32 - drop pointless static qualifier in stm32_hash_remove()
  crypto: chelsio - Fixed Traffic Stall
  crypto: marvell - Remove set but not used variable 'ivsize'
  crypto: ccp - Update driver messages to remove some confusion
  crypto: adiantum - add 1536 and 4096-byte test vectors
  crypto: nhpoly1305 - add a test vector with len % 16 != 0
  crypto: arm/aes-ce - update IV after partial final CTR block
  ...
hifive-unleashed-5.1
Linus Torvalds 2019-03-05 09:09:55 -08:00
parent 6456300356 0918f18c71
commit 63bdf4284c
170 changed files with 8942 additions and 11459 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

19
Documentation/devicetree/bindings/crypto/samsung-slimsss.txt 100644
View File

@ -0,0 +1,19 @@
Samsung SoC SlimSSS (Slim Security SubSystem) module
The SlimSSS module in Exynos5433 SoC supports the following:
-- Feeder (FeedCtrl)
-- Advanced Encryption Standard (AES) with ECB,CBC,CTR,XTS and (CBC/XTS)/CTS
-- SHA-1/SHA-256 and (SHA-1/SHA-256)/HMAC
Required properties:
- compatible : Should contain entry for slimSSS version:
- "samsung,exynos5433-slim-sss" for Exynos5433 SoC.
- reg : Offset and length of the register set for the module
- interrupts : interrupt specifiers of SlimSSS module interrupts (one feed
control interrupt).
- clocks : list of clock phandle and specifier pairs for all clocks listed in
clock-names property.
- clock-names : list of device clock input names; should contain "pclk" and
"aclk" for slim-sss in Exynos5433.

1
MAINTAINERS
View File

@ -3529,7 +3529,6 @@ F: include/linux/spi/cc2520.h
F: Documentation/devicetree/bindings/net/ieee802154/cc2520.txt F: Documentation/devicetree/bindings/net/ieee802154/cc2520.txt
CCREE ARM TRUSTZONE CRYPTOCELL REE DRIVER CCREE ARM TRUSTZONE CRYPTOCELL REE DRIVER
M: Yael Chemla <yael.chemla@foss.arm.com>
M: Gilad Ben-Yossef <gilad@benyossef.com> M: Gilad Ben-Yossef <gilad@benyossef.com>
L: linux-crypto@vger.kernel.org L: linux-crypto@vger.kernel.org
S: Supported S: Supported

26
arch/arm/crypto/aes-ce-core.S
View File

@ -317,25 +317,27 @@ ENTRY(ce_aes_ctr_encrypt)
.Lctrloop: .Lctrloop:
vmov q0, q6 vmov q0, q6
bl aes_encrypt bl aes_encrypt
subs r4, r4, #1
bmi .Lctrtailblock @ blocks < 0 means tail block
vld1.8 {q3}, [r1]!
veor q3, q0, q3
vst1.8 {q3}, [r0]!
adds r6, r6, #1 @ increment BE ctr adds r6, r6, #1 @ increment BE ctr
rev ip, r6 rev ip, r6
vmov s27, ip vmov s27, ip
bcs .Lctrcarry bcs .Lctrcarry
teq r4, #0
.Lctrcarrydone:
subs r4, r4, #1
bmi .Lctrtailblock @ blocks < 0 means tail block
vld1.8 {q3}, [r1]!
veor q3, q0, q3
vst1.8 {q3}, [r0]!
bne .Lctrloop bne .Lctrloop
.Lctrout: .Lctrout:
vst1.8 {q6}, [r5] vst1.8 {q6}, [r5] @ return next CTR value
pop {r4-r6, pc} pop {r4-r6, pc}
.Lctrtailblock: .Lctrtailblock:
vst1.8 {q0}, [r0, :64] @ return just the key stream vst1.8 {q0}, [r0, :64] @ return the key stream
pop {r4-r6, pc} b .Lctrout
.Lctrcarry: .Lctrcarry:
.irp sreg, s26, s25, s24 .irp sreg, s26, s25, s24
@ -344,11 +346,9 @@ ENTRY(ce_aes_ctr_encrypt)
adds ip, ip, #1 adds ip, ip, #1
rev ip, ip rev ip, ip
vmov \sreg, ip vmov \sreg, ip
bcc 0f bcc .Lctrcarrydone
.endr .endr
0: teq r4, #0 b .Lctrcarrydone
beq .Lctrout
b .Lctrloop
ENDPROC(ce_aes_ctr_encrypt) ENDPROC(ce_aes_ctr_encrypt)
/* /*

524
arch/arm/crypto/crct10dif-ce-core.S
View File

@ -2,12 +2,14 @@
// Accelerated CRC-T10DIF using ARM NEON and Crypto Extensions instructions // Accelerated CRC-T10DIF using ARM NEON and Crypto Extensions instructions
// //
// Copyright (C) 2016 Linaro Ltd <ard.biesheuvel@linaro.org> // Copyright (C) 2016 Linaro Ltd <ard.biesheuvel@linaro.org>
// Copyright (C) 2019 Google LLC <ebiggers@google.com>
// //
// This program is free software; you can redistribute it and/or modify // This program is free software; you can redistribute it and/or modify
// it under the terms of the GNU General Public License version 2 as // it under the terms of the GNU General Public License version 2 as
// published by the Free Software Foundation. // published by the Free Software Foundation.
// //
// Derived from the x86 version:
// //
// Implement fast CRC-T10DIF computation with SSE and PCLMULQDQ instructions // Implement fast CRC-T10DIF computation with SSE and PCLMULQDQ instructions
// //
@ -54,19 +56,11 @@
// NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS // NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
// SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. // SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
// //
// Function API:
// UINT16 crc_t10dif_pcl(
// UINT16 init_crc, //initial CRC value, 16 bits
// const unsigned char *buf, //buffer pointer to calculate CRC on
// UINT64 len //buffer length in bytes (64-bit data)
// );
//
// Reference paper titled "Fast CRC Computation for Generic // Reference paper titled "Fast CRC Computation for Generic
// Polynomials Using PCLMULQDQ Instruction" // Polynomials Using PCLMULQDQ Instruction"
// URL: http://www.intel.com/content/dam/www/public/us/en/documents // URL: http://www.intel.com/content/dam/www/public/us/en/documents
// /white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf // /white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf
// //
//
#include <linux/linkage.h> #include <linux/linkage.h>
#include <asm/assembler.h> #include <asm/assembler.h>
@ -78,13 +72,14 @@
#endif #endif
.text .text
.arch armv7-a
.fpu crypto-neon-fp-armv8 .fpu crypto-neon-fp-armv8
arg1_low32 .req r0 init_crc .req r0
arg2 .req r1 buf .req r1
arg3 .req r2 len .req r2
qzr .req q13 fold_consts_ptr .req ip
q0l .req d0 q0l .req d0
q0h .req d1 q0h .req d1
@ -102,82 +97,35 @@
q6h .req d13 q6h .req d13
q7l .req d14 q7l .req d14
q7h .req d15 q7h .req d15
q8l .req d16
q8h .req d17
q9l .req d18
q9h .req d19
q10l .req d20
q10h .req d21
q11l .req d22
q11h .req d23
q12l .req d24
q12h .req d25
ENTRY(crc_t10dif_pmull) FOLD_CONSTS .req q10
vmov.i8 qzr, #0 // init zero register FOLD_CONST_L .req q10l
FOLD_CONST_H .req q10h
// adjust the 16-bit initial_crc value, scale it to 32 bits // Fold reg1, reg2 into the next 32 data bytes, storing the result back
lsl arg1_low32, arg1_low32, #16 // into reg1, reg2.
.macro fold_32_bytes, reg1, reg2
vld1.64 {q11-q12}, [buf]!
// check if smaller than 256 vmull.p64 q8, \reg1\()h, FOLD_CONST_H
cmp arg3, #256 vmull.p64 \reg1, \reg1\()l, FOLD_CONST_L
vmull.p64 q9, \reg2\()h, FOLD_CONST_H
vmull.p64 \reg2, \reg2\()l, FOLD_CONST_L
// for sizes less than 128, we can't fold 64B at a time... CPU_LE( vrev64.8 q11, q11 )
blt _less_than_128 CPU_LE( vrev64.8 q12, q12 )
vswp q11l, q11h
// load the initial crc value vswp q12l, q12h
// crc value does not need to be byte-reflected, but it needs
// to be moved to the high part of the register.
// because data will be byte-reflected and will align with
// initial crc at correct place.
vmov s0, arg1_low32 // initial crc
vext.8 q10, qzr, q0, #4
// receive the initial 64B data, xor the initial crc value
vld1.64 {q0-q1}, [arg2, :128]!
vld1.64 {q2-q3}, [arg2, :128]!
vld1.64 {q4-q5}, [arg2, :128]!
vld1.64 {q6-q7}, [arg2, :128]!
CPU_LE( vrev64.8 q0, q0 )
CPU_LE( vrev64.8 q1, q1 )
CPU_LE( vrev64.8 q2, q2 )
CPU_LE( vrev64.8 q3, q3 )
CPU_LE( vrev64.8 q4, q4 )
CPU_LE( vrev64.8 q5, q5 )
CPU_LE( vrev64.8 q6, q6 )
CPU_LE( vrev64.8 q7, q7 )
vswp d0, d1
vswp d2, d3
vswp d4, d5
vswp d6, d7
vswp d8, d9
vswp d10, d11
vswp d12, d13
vswp d14, d15
// XOR the initial_crc value
veor.8 q0, q0, q10
adr ip, rk3
vld1.64 {q10}, [ip, :128] // xmm10 has rk3 and rk4
//
// we subtract 256 instead of 128 to save one instruction from the loop
//
sub arg3, arg3, #256
// at this section of the code, there is 64*x+y (0<=y<64) bytes of
// buffer. The _fold_64_B_loop will fold 64B at a time
// until we have 64+y Bytes of buffer
// fold 64B at a time. This section of the code folds 4 vector
// registers in parallel
_fold_64_B_loop:
.macro fold64, reg1, reg2
vld1.64 {q11-q12}, [arg2, :128]!
vmull.p64 q8, \reg1\()h, d21
vmull.p64 \reg1, \reg1\()l, d20
vmull.p64 q9, \reg2\()h, d21
vmull.p64 \reg2, \reg2\()l, d20
CPU_LE( vrev64.8 q11, q11 )
CPU_LE( vrev64.8 q12, q12 )
vswp d22, d23
vswp d24, d25
veor.8 \reg1, \reg1, q8 veor.8 \reg1, \reg1, q8
veor.8 \reg2, \reg2, q9 veor.8 \reg2, \reg2, q9
@ -185,242 +133,248 @@ CPU_LE( vrev64.8 q12, q12 )
veor.8 \reg2, \reg2, q12 veor.8 \reg2, \reg2, q12
.endm .endm
fold64 q0, q1 // Fold src_reg into dst_reg, optionally loading the next fold constants
fold64 q2, q3 .macro fold_16_bytes, src_reg, dst_reg, load_next_consts
fold64 q4, q5 vmull.p64 q8, \src_reg\()l, FOLD_CONST_L
fold64 q6, q7 vmull.p64 \src_reg, \src_reg\()h, FOLD_CONST_H
.ifnb \load_next_consts
subs arg3, arg3, #128 vld1.64 {FOLD_CONSTS}, [fold_consts_ptr, :128]!
// check if there is another 64B in the buffer to be able to fold
bge _fold_64_B_loop
// at this point, the buffer pointer is pointing at the last y Bytes
// of the buffer the 64B of folded data is in 4 of the vector
// registers: v0, v1, v2, v3
// fold the 8 vector registers to 1 vector register with different
// constants
adr ip, rk9
vld1.64 {q10}, [ip, :128]!
.macro fold16, reg, rk
vmull.p64 q8, \reg\()l, d20
vmull.p64 \reg, \reg\()h, d21
.ifnb \rk
vld1.64 {q10}, [ip, :128]!
.endif .endif
veor.8 q7, q7, q8 veor.8 \dst_reg, \dst_reg, q8
veor.8 q7, q7, \reg veor.8 \dst_reg, \dst_reg, \src_reg
.endm .endm
fold16 q0, rk11 .macro __adrl, out, sym
fold16 q1, rk13 movw \out, #:lower16:\sym
fold16 q2, rk15 movt \out, #:upper16:\sym
fold16 q3, rk17 .endm
fold16 q4, rk19
fold16 q5, rk1
fold16 q6
// instead of 64, we add 48 to the loop counter to save 1 instruction //
// from the loop instead of a cmp instruction, we use the negative // u16 crc_t10dif_pmull(u16 init_crc, const u8 *buf, size_t len);
// flag with the jl instruction //
adds arg3, arg3, #(128-16) // Assumes len >= 16.
blt _final_reduction_for_128 //
ENTRY(crc_t10dif_pmull)
// now we have 16+y bytes left to reduce. 16 Bytes is in register v7 // For sizes less than 256 bytes, we can't fold 128 bytes at a time.
// and the rest is in memory. We can fold 16 bytes at a time if y>=16 cmp len, #256
// continue folding 16B at a time blt .Lless_than_256_bytes
_16B_reduction_loop: __adrl fold_consts_ptr, .Lfold_across_128_bytes_consts
vmull.p64 q8, d14, d20
vmull.p64 q7, d15, d21 // Load the first 128 data bytes. Byte swapping is necessary to make
// the bit order match the polynomial coefficient order.
vld1.64 {q0-q1}, [buf]!
vld1.64 {q2-q3}, [buf]!
vld1.64 {q4-q5}, [buf]!
vld1.64 {q6-q7}, [buf]!
CPU_LE( vrev64.8 q0, q0 )
CPU_LE( vrev64.8 q1, q1 )
CPU_LE( vrev64.8 q2, q2 )
CPU_LE( vrev64.8 q3, q3 )
CPU_LE( vrev64.8 q4, q4 )
CPU_LE( vrev64.8 q5, q5 )
CPU_LE( vrev64.8 q6, q6 )
CPU_LE( vrev64.8 q7, q7 )
vswp q0l, q0h
vswp q1l, q1h
vswp q2l, q2h
vswp q3l, q3h
vswp q4l, q4h
vswp q5l, q5h
vswp q6l, q6h
vswp q7l, q7h
// XOR the first 16 data *bits* with the initial CRC value.
vmov.i8 q8h, #0
vmov.u16 q8h[3], init_crc
veor q0h, q0h, q8h
// Load the constants for folding across 128 bytes.
vld1.64 {FOLD_CONSTS}, [fold_consts_ptr, :128]!
// Subtract 128 for the 128 data bytes just consumed. Subtract another
// 128 to simplify the termination condition of the following loop.
sub len, len, #256
// While >= 128 data bytes remain (not counting q0-q7), fold the 128
// bytes q0-q7 into them, storing the result back into q0-q7.
.Lfold_128_bytes_loop:
fold_32_bytes q0, q1
fold_32_bytes q2, q3
fold_32_bytes q4, q5
fold_32_bytes q6, q7
subs len, len, #128
bge .Lfold_128_bytes_loop
// Now fold the 112 bytes in q0-q6 into the 16 bytes in q7.
// Fold across 64 bytes.
vld1.64 {FOLD_CONSTS}, [fold_consts_ptr, :128]!
fold_16_bytes q0, q4
fold_16_bytes q1, q5
fold_16_bytes q2, q6
fold_16_bytes q3, q7, 1
// Fold across 32 bytes.
fold_16_bytes q4, q6
fold_16_bytes q5, q7, 1
// Fold across 16 bytes.
fold_16_bytes q6, q7
// Add 128 to get the correct number of data bytes remaining in 0...127
// (not counting q7), following the previous extra subtraction by 128.
// Then subtract 16 to simplify the termination condition of the
// following loop.
adds len, len, #(128-16)
// While >= 16 data bytes remain (not counting q7), fold the 16 bytes q7
// into them, storing the result back into q7.
blt .Lfold_16_bytes_loop_done
.Lfold_16_bytes_loop:
vmull.p64 q8, q7l, FOLD_CONST_L
vmull.p64 q7, q7h, FOLD_CONST_H
veor.8 q7, q7, q8 veor.8 q7, q7, q8
vld1.64 {q0}, [buf]!
vld1.64 {q0}, [arg2, :128]! CPU_LE( vrev64.8 q0, q0 )
CPU_LE( vrev64.8 q0, q0 ) vswp q0l, q0h
vswp d0, d1
veor.8 q7, q7, q0 veor.8 q7, q7, q0
subs arg3, arg3, #16 subs len, len, #16
bge .Lfold_16_bytes_loop
// instead of a cmp instruction, we utilize the flags with the .Lfold_16_bytes_loop_done:
// jge instruction equivalent of: cmp arg3, 16-16 // Add 16 to get the correct number of data bytes remaining in 0...15
// check if there is any more 16B in the buffer to be able to fold // (not counting q7), following the previous extra subtraction by 16.
bge _16B_reduction_loop adds len, len, #16
beq .Lreduce_final_16_bytes
// now we have 16+z bytes left to reduce, where 0<= z < 16. .Lhandle_partial_segment:
// first, we reduce the data in the xmm7 register // Reduce the last '16 + len' bytes where 1 <= len <= 15 and the first
// 16 bytes are in q7 and the rest are the remaining data in 'buf'. To
// do this without needing a fold constant for each possible 'len',
// redivide the bytes into a first chunk of 'len' bytes and a second
// chunk of 16 bytes, then fold the first chunk into the second.
_final_reduction_for_128: // q0 = last 16 original data bytes
// check if any more data to fold. If not, compute the CRC of add buf, buf, len
// the final 128 bits sub buf, buf, #16
adds arg3, arg3, #16 vld1.64 {q0}, [buf]
beq _128_done CPU_LE( vrev64.8 q0, q0 )
vswp q0l, q0h
// here we are getting data that is less than 16 bytes. // q1 = high order part of second chunk: q7 left-shifted by 'len' bytes.
// since we know that there was data before the pointer, we can __adrl r3, .Lbyteshift_table + 16
// offset the input pointer before the actual point, to receive sub r3, r3, len
// exactly 16 bytes. after that the registers need to be adjusted. vld1.8 {q2}, [r3]
_get_last_two_regs: vtbl.8 q1l, {q7l-q7h}, q2l
add arg2, arg2, arg3 vtbl.8 q1h, {q7l-q7h}, q2h
sub arg2, arg2, #16
vld1.64 {q1}, [arg2]
CPU_LE( vrev64.8 q1, q1 )
vswp d2, d3
// get rid of the extra data that was loaded before // q3 = first chunk: q7 right-shifted by '16-len' bytes.
// load the shift constant vmov.i8 q3, #0x80
adr ip, tbl_shf_table + 16 veor.8 q2, q2, q3
sub ip, ip, arg3 vtbl.8 q3l, {q7l-q7h}, q2l
vld1.8 {q0}, [ip] vtbl.8 q3h, {q7l-q7h}, q2h
// shift v2 to the left by arg3 bytes // Convert to 8-bit masks: 'len' 0x00 bytes, then '16-len' 0xff bytes.
vtbl.8 d4, {d14-d15}, d0 vshr.s8 q2, q2, #7
vtbl.8 d5, {d14-d15}, d1
// shift v7 to the right by 16-arg3 bytes // q2 = second chunk: 'len' bytes from q0 (low-order bytes),
vmov.i8 q9, #0x80 // then '16-len' bytes from q1 (high-order bytes).
veor.8 q0, q0, q9 vbsl.8 q2, q1, q0
vtbl.8 d18, {d14-d15}, d0
vtbl.8 d19, {d14-d15}, d1
// blend // Fold the first chunk into the second chunk, storing the result in q7.
vshr.s8 q0, q0, #7 // convert to 8-bit mask vmull.p64 q0, q3l, FOLD_CONST_L
vbsl.8 q0, q2, q1 vmull.p64 q7, q3h, FOLD_CONST_H
// fold 16 Bytes
vmull.p64 q8, d18, d20
vmull.p64 q7, d19, d21
veor.8 q7, q7, q8
veor.8 q7, q7, q0 veor.8 q7, q7, q0
veor.8 q7, q7, q2
_128_done: .Lreduce_final_16_bytes:
// compute crc of a 128-bit value // Reduce the 128-bit value M(x), stored in q7, to the final 16-bit CRC.
vldr d20, rk5
vldr d21, rk6 // rk5 and rk6 in xmm10
// 64b fold // Load 'x^48 * (x^48 mod G(x))' and 'x^48 * (x^80 mod G(x))'.
vext.8 q0, qzr, q7, #8 vld1.64 {FOLD_CONSTS}, [fold_consts_ptr, :128]!
vmull.p64 q7, d15, d20
veor.8 q7, q7, q0
// 32b fold // Fold the high 64 bits into the low 64 bits, while also multiplying by
vext.8 q0, q7, qzr, #12 // x^64. This produces a 128-bit value congruent to x^64 * M(x) and
vmov s31, s3 // whose low 48 bits are 0.
vmull.p64 q0, d0, d21 vmull.p64 q0, q7h, FOLD_CONST_H // high bits * x^48 * (x^80 mod G(x))
veor.8 q7, q0, q7 veor.8 q0h, q0h, q7l // + low bits * x^64
// barrett reduction // Fold the high 32 bits into the low 96 bits. This produces a 96-bit
_barrett: // value congruent to x^64 * M(x) and whose low 48 bits are 0.
vldr d20, rk7 vmov.i8 q1, #0
vldr d21, rk8 vmov s4, s3 // extract high 32 bits
vmov s3, s5 // zero high 32 bits
vmull.p64 q1, q1l, FOLD_CONST_L // high 32 bits * x^48 * (x^48 mod G(x))
veor.8 q0, q0, q1 // + low bits
vmull.p64 q0, d15, d20 // Load G(x) and floor(x^48 / G(x)).
vext.8 q0, qzr, q0, #12 vld1.64 {FOLD_CONSTS}, [fold_consts_ptr, :128]
vmull.p64 q0, d1, d21
vext.8 q0, qzr, q0, #12
veor.8 q7, q7, q0
vmov r0, s29
_cleanup: // Use Barrett reduction to compute the final CRC value.
// scale the result back to 16 bits vmull.p64 q1, q0h, FOLD_CONST_H // high 32 bits * floor(x^48 / G(x))
lsr r0, r0, #16 vshr.u64 q1l, q1l, #32 // /= x^32
vmull.p64 q1, q1l, FOLD_CONST_L // *= G(x)
vshr.u64 q0l, q0l, #48
veor.8 q0l, q0l, q1l // + low 16 nonzero bits
// Final CRC value (x^16 * M(x)) mod G(x) is in low 16 bits of q0.
vmov.u16 r0, q0l[0]
bx lr bx lr
_less_than_128: .Lless_than_256_bytes:
teq arg3, #0 // Checksumming a buffer of length 16...255 bytes
beq _cleanup
vmov.i8 q0, #0 __adrl fold_consts_ptr, .Lfold_across_16_bytes_consts
vmov s3, arg1_low32 // get the initial crc value
vld1.64 {q7}, [arg2, :128]! // Load the first 16 data bytes.
CPU_LE( vrev64.8 q7, q7 ) vld1.64 {q7}, [buf]!
vswp d14, d15 CPU_LE( vrev64.8 q7, q7 )
veor.8 q7, q7, q0 vswp q7l, q7h
cmp arg3, #16 // XOR the first 16 data *bits* with the initial CRC value.
beq _128_done // exactly 16 left vmov.i8 q0h, #0
blt _less_than_16_left vmov.u16 q0h[3], init_crc
veor.8 q7h, q7h, q0h
// now if there is, load the constants // Load the fold-across-16-bytes constants.
vldr d20, rk1 vld1.64 {FOLD_CONSTS}, [fold_consts_ptr, :128]!
vldr d21, rk2 // rk1 and rk2 in xmm10
// check if there is enough buffer to be able to fold 16B at a time cmp len, #16
subs arg3, arg3, #32 beq .Lreduce_final_16_bytes // len == 16
addlt arg3, arg3, #16 subs len, len, #32
blt _get_last_two_regs addlt len, len, #16
b _16B_reduction_loop blt .Lhandle_partial_segment // 17 <= len <= 31
b .Lfold_16_bytes_loop // 32 <= len <= 255
_less_than_16_left:
// shl r9, 4
adr ip, tbl_shf_table + 16
sub ip, ip, arg3
vld1.8 {q0}, [ip]
vmov.i8 q9, #0x80
veor.8 q0, q0, q9
vtbl.8 d18, {d14-d15}, d0
vtbl.8 d15, {d14-d15}, d1
vmov d14, d18
b _128_done
ENDPROC(crc_t10dif_pmull) ENDPROC(crc_t10dif_pmull)
// precomputed constants .section ".rodata", "a"
// these constants are precomputed from the poly:
// 0x8bb70000 (0x8bb7 scaled to 32 bits)
.align 4 .align 4
// Q = 0x18BB70000
// rk1 = 2^(32*3) mod Q << 32
// rk2 = 2^(32*5) mod Q << 32
// rk3 = 2^(32*15) mod Q << 32
// rk4 = 2^(32*17) mod Q << 32
// rk5 = 2^(32*3) mod Q << 32
// rk6 = 2^(32*2) mod Q << 32
// rk7 = floor(2^64/Q)
// rk8 = Q
rk3: .quad 0x9d9d000000000000 // Fold constants precomputed from the polynomial 0x18bb7
rk4: .quad 0x7cf5000000000000 // G(x) = x^16 + x^15 + x^11 + x^9 + x^8 + x^7 + x^5 + x^4 + x^2 + x^1 + x^0
rk5: .quad 0x2d56000000000000 .Lfold_across_128_bytes_consts:
rk6: .quad 0x1368000000000000 .quad 0x0000000000006123 // x^(8*128) mod G(x)
rk7: .quad 0x00000001f65a57f8 .quad 0x0000000000002295 // x^(8*128+64) mod G(x)
rk8: .quad 0x000000018bb70000 // .Lfold_across_64_bytes_consts:
rk9: .quad 0xceae000000000000 .quad 0x0000000000001069 // x^(4*128) mod G(x)
rk10: .quad 0xbfd6000000000000 .quad 0x000000000000dd31 // x^(4*128+64) mod G(x)
rk11: .quad 0x1e16000000000000 // .Lfold_across_32_bytes_consts:
rk12: .quad 0x713c000000000000 .quad 0x000000000000857d // x^(2*128) mod G(x)
rk13: .quad 0xf7f9000000000000 .quad 0x0000000000007acc // x^(2*128+64) mod G(x)
rk14: .quad 0x80a6000000000000 .Lfold_across_16_bytes_consts:
rk15: .quad 0x044c000000000000 .quad 0x000000000000a010 // x^(1*128) mod G(x)
rk16: .quad 0xe658000000000000 .quad 0x0000000000001faa // x^(1*128+64) mod G(x)
rk17: .quad 0xad18000000000000 // .Lfinal_fold_consts:
rk18: .quad 0xa497000000000000 .quad 0x1368000000000000 // x^48 * (x^48 mod G(x))
rk19: .quad 0x6ee3000000000000 .quad 0x2d56000000000000 // x^48 * (x^80 mod G(x))
rk20: .quad 0xe7b5000000000000 // .Lbarrett_reduction_consts:
rk1: .quad 0x2d56000000000000 .quad 0x0000000000018bb7 // G(x)
rk2: .quad 0x06df000000000000 .quad 0x00000001f65a57f8 // floor(x^48 / G(x))
tbl_shf_table:
// use these values for shift constants for the tbl/tbx instruction
// different alignments result in values as shown:
// DDQ 0x008f8e8d8c8b8a898887868584838281 # shl 15 (16-1) / shr1
// DDQ 0x01008f8e8d8c8b8a8988878685848382 # shl 14 (16-3) / shr2
// DDQ 0x0201008f8e8d8c8b8a89888786858483 # shl 13 (16-4) / shr3
// DDQ 0x030201008f8e8d8c8b8a898887868584 # shl 12 (16-4) / shr4
// DDQ 0x04030201008f8e8d8c8b8a8988878685 # shl 11 (16-5) / shr5
// DDQ 0x0504030201008f8e8d8c8b8a89888786 # shl 10 (16-6) / shr6
// DDQ 0x060504030201008f8e8d8c8b8a898887 # shl 9 (16-7) / shr7
// DDQ 0x07060504030201008f8e8d8c8b8a8988 # shl 8 (16-8) / shr8
// DDQ 0x0807060504030201008f8e8d8c8b8a89 # shl 7 (16-9) / shr9
// DDQ 0x090807060504030201008f8e8d8c8b8a # shl 6 (16-10) / shr10
// DDQ 0x0a090807060504030201008f8e8d8c8b # shl 5 (16-11) / shr11
// DDQ 0x0b0a090807060504030201008f8e8d8c # shl 4 (16-12) / shr12
// DDQ 0x0c0b0a090807060504030201008f8e8d # shl 3 (16-13) / shr13
// DDQ 0x0d0c0b0a090807060504030201008f8e # shl 2 (16-14) / shr14
// DDQ 0x0e0d0c0b0a090807060504030201008f # shl 1 (16-15) / shr15
// For 1 <= len <= 15, the 16-byte vector beginning at &byteshift_table[16 -
// len] is the index vector to shift left by 'len' bytes, and is also {0x80,
// ..., 0x80} XOR the index vector to shift right by '16 - len' bytes.
.Lbyteshift_table:
.byte 0x0, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87 .byte 0x0, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87
.byte 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f .byte 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f
.byte 0x0, 0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7 .byte 0x0, 0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7

25
arch/arm/crypto/crct10dif-ce-glue.c
View File

@ -21,7 +21,7 @@
#define CRC_T10DIF_PMULL_CHUNK_SIZE 16U #define CRC_T10DIF_PMULL_CHUNK_SIZE 16U
asmlinkage u16 crc_t10dif_pmull(u16 init_crc, const u8 buf[], u32 len); asmlinkage u16 crc_t10dif_pmull(u16 init_crc, const u8 *buf, size_t len);
static int crct10dif_init(struct shash_desc *desc) static int crct10dif_init(struct shash_desc *desc)
{ {
@ -35,26 +35,15 @@ static int crct10dif_update(struct shash_desc *desc, const u8 *data,
unsigned int length) unsigned int length)
{ {
u16 *crc = shash_desc_ctx(desc); u16 *crc = shash_desc_ctx(desc);
unsigned int l;
if (!may_use_simd()) { if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) {
*crc = crc_t10dif_generic(*crc, data, length); kernel_neon_begin();
*crc = crc_t10dif_pmull(*crc, data, length);
kernel_neon_end();
} else { } else {
if (unlikely((u32)data % CRC_T10DIF_PMULL_CHUNK_SIZE)) { *crc = crc_t10dif_generic(*crc, data, length);
l = min_t(u32, length, CRC_T10DIF_PMULL_CHUNK_SIZE -
((u32)data % CRC_T10DIF_PMULL_CHUNK_SIZE));
*crc = crc_t10dif_generic(*crc, data, l);
length -= l;
data += l;
}
if (length > 0) {
kernel_neon_begin();
*crc = crc_t10dif_pmull(*crc, data, length);
kernel_neon_end();
}
} }
return 0; return 0;
} }

5
arch/arm64/crypto/aes-ce-ccm-core.S
View File

@ -74,12 +74,13 @@ ENTRY(ce_aes_ccm_auth_data)
beq 10f beq 10f
ext v0.16b, v0.16b, v0.16b, #1 /* rotate out the mac bytes */ ext v0.16b, v0.16b, v0.16b, #1 /* rotate out the mac bytes */
b 7b b 7b
8: mov w7, w8 8: cbz w8, 91f
mov w7, w8
add w8, w8, #16 add w8, w8, #16
9: ext v1.16b, v1.16b, v1.16b, #1 9: ext v1.16b, v1.16b, v1.16b, #1
adds w7, w7, #1 adds w7, w7, #1
bne 9b bne 9b
eor v0.16b, v0.16b, v1.16b 91: eor v0.16b, v0.16b, v1.16b
st1 {v0.16b}, [x0] st1 {v0.16b}, [x0]
10: str w8, [x3] 10: str w8, [x3]
ret ret

8
arch/arm64/crypto/aes-ce-ccm-glue.c
View File

@ -125,7 +125,7 @@ static void ccm_update_mac(struct crypto_aes_ctx *key, u8 mac[], u8 const in[],
abytes -= added; abytes -= added;
} }
while (abytes > AES_BLOCK_SIZE) { while (abytes >= AES_BLOCK_SIZE) {
__aes_arm64_encrypt(key->key_enc, mac, mac, __aes_arm64_encrypt(key->key_enc, mac, mac,
num_rounds(key)); num_rounds(key));
crypto_xor(mac, in, AES_BLOCK_SIZE); crypto_xor(mac, in, AES_BLOCK_SIZE);
@ -139,8 +139,6 @@ static void ccm_update_mac(struct crypto_aes_ctx *key, u8 mac[], u8 const in[],
num_rounds(key)); num_rounds(key));
crypto_xor(mac, in, abytes); crypto_xor(mac, in, abytes);
*macp = abytes; *macp = abytes;
} else {
*macp = 0;
} }
} }
} }
@ -255,7 +253,7 @@ static int ccm_encrypt(struct aead_request *req)
/* preserve the original iv for the final round */ /* preserve the original iv for the final round */
memcpy(buf, req->iv, AES_BLOCK_SIZE); memcpy(buf, req->iv, AES_BLOCK_SIZE);
err = skcipher_walk_aead_encrypt(&walk, req, true); err = skcipher_walk_aead_encrypt(&walk, req, false);
if (may_use_simd()) { if (may_use_simd()) {
while (walk.nbytes) { while (walk.nbytes) {
@ -313,7 +311,7 @@ static int ccm_decrypt(struct aead_request *req)
/* preserve the original iv for the final round */ /* preserve the original iv for the final round */
memcpy(buf, req->iv, AES_BLOCK_SIZE); memcpy(buf, req->iv, AES_BLOCK_SIZE);
err = skcipher_walk_aead_decrypt(&walk, req, true); err = skcipher_walk_aead_decrypt(&walk, req, false);
if (may_use_simd()) { if (may_use_simd()) {
while (walk.nbytes) { while (walk.nbytes) {

3
arch/arm64/crypto/aes-modes.S
View File

@ -320,8 +320,7 @@ AES_ENTRY(aes_ctr_encrypt)
.Lctrtailblock: .Lctrtailblock:
st1 {v0.16b}, [x0] st1 {v0.16b}, [x0]
ldp x29, x30, [sp], #16 b .Lctrout
ret
.Lctrcarry: .Lctrcarry:
umov x7, v4.d[0] /* load upper word of ctr */ umov x7, v4.d[0] /* load upper word of ctr */

8
arch/arm64/crypto/aes-neonbs-core.S
View File

@ -971,18 +971,22 @@ CPU_LE( rev x8, x8 )
8: next_ctr v0 8: next_ctr v0
st1 {v0.16b}, [x24] st1 {v0.16b}, [x24]
cbz x23, 0f cbz x23, .Lctr_done
cond_yield_neon 98b cond_yield_neon 98b
b 99b b 99b
0: frame_pop .Lctr_done:
frame_pop
ret ret
/* /*
* If we are handling the tail of the input (x6 != NULL), return the * If we are handling the tail of the input (x6 != NULL), return the
* final keystream block back to the caller. * final keystream block back to the caller.
*/ */
0: cbz x25, 8b
st1 {v0.16b}, [x25]
b 8b
1: cbz x25, 8b 1: cbz x25, 8b
st1 {v1.16b}, [x25] st1 {v1.16b}, [x25]
b 8b b 8b

477
arch/arm64/crypto/crct10dif-ce-core.S
View File

@ -2,12 +2,14 @@
// Accelerated CRC-T10DIF using arm64 NEON and Crypto Extensions instructions // Accelerated CRC-T10DIF using arm64 NEON and Crypto Extensions instructions
// //
// Copyright (C) 2016 Linaro Ltd <ard.biesheuvel@linaro.org> // Copyright (C) 2016 Linaro Ltd <ard.biesheuvel@linaro.org>
// Copyright (C) 2019 Google LLC <ebiggers@google.com>
// //
// This program is free software; you can redistribute it and/or modify // This program is free software; you can redistribute it and/or modify
// it under the terms of the GNU General Public License version 2 as // it under the terms of the GNU General Public License version 2 as
// published by the Free Software Foundation. // published by the Free Software Foundation.
// //
// Derived from the x86 version:
// //
// Implement fast CRC-T10DIF computation with SSE and PCLMULQDQ instructions // Implement fast CRC-T10DIF computation with SSE and PCLMULQDQ instructions
// //
@ -54,19 +56,11 @@
// NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS // NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
// SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. // SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
// //
// Function API:
// UINT16 crc_t10dif_pcl(
// UINT16 init_crc, //initial CRC value, 16 bits
// const unsigned char *buf, //buffer pointer to calculate CRC on
// UINT64 len //buffer length in bytes (64-bit data)
// );
//
// Reference paper titled "Fast CRC Computation for Generic // Reference paper titled "Fast CRC Computation for Generic
// Polynomials Using PCLMULQDQ Instruction" // Polynomials Using PCLMULQDQ Instruction"
// URL: http://www.intel.com/content/dam/www/public/us/en/documents // URL: http://www.intel.com/content/dam/www/public/us/en/documents
// /white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf // /white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf
// //
//
#include <linux/linkage.h> #include <linux/linkage.h>
#include <asm/assembler.h> #include <asm/assembler.h>
@ -74,14 +68,14 @@
.text .text
.cpu generic+crypto .cpu generic+crypto
arg1_low32 .req w19 init_crc .req w19
arg2 .req x20 buf .req x20
arg3 .req x21 len .req x21
fold_consts_ptr .req x22
vzr .req v13 fold_consts .req v10
ad .req v14 ad .req v14
bd .req v10
k00_16 .req v15 k00_16 .req v15
k32_48 .req v16 k32_48 .req v16
@ -143,11 +137,11 @@ __pmull_p8_core:
ext t5.8b, ad.8b, ad.8b, #2 // A2 ext t5.8b, ad.8b, ad.8b, #2 // A2
ext t6.8b, ad.8b, ad.8b, #3 // A3 ext t6.8b, ad.8b, ad.8b, #3 // A3
pmull t4.8h, t4.8b, bd.8b // F = A1*B pmull t4.8h, t4.8b, fold_consts.8b // F = A1*B
pmull t8.8h, ad.8b, bd1.8b // E = A*B1 pmull t8.8h, ad.8b, bd1.8b // E = A*B1
pmull t5.8h, t5.8b, bd.8b // H = A2*B pmull t5.8h, t5.8b, fold_consts.8b // H = A2*B
pmull t7.8h, ad.8b, bd2.8b // G = A*B2 pmull t7.8h, ad.8b, bd2.8b // G = A*B2
pmull t6.8h, t6.8b, bd.8b // J = A3*B pmull t6.8h, t6.8b, fold_consts.8b // J = A3*B
pmull t9.8h, ad.8b, bd3.8b // I = A*B3 pmull t9.8h, ad.8b, bd3.8b // I = A*B3
pmull t3.8h, ad.8b, bd4.8b // K = A*B4 pmull t3.8h, ad.8b, bd4.8b // K = A*B4
b 0f b 0f
@ -157,11 +151,11 @@ __pmull_p8_core:
tbl t5.16b, {ad.16b}, perm2.16b // A2 tbl t5.16b, {ad.16b}, perm2.16b // A2
tbl t6.16b, {ad.16b}, perm3.16b // A3 tbl t6.16b, {ad.16b}, perm3.16b // A3
pmull2 t4.8h, t4.16b, bd.16b // F = A1*B pmull2 t4.8h, t4.16b, fold_consts.16b // F = A1*B
pmull2 t8.8h, ad.16b, bd1.16b // E = A*B1 pmull2 t8.8h, ad.16b, bd1.16b // E = A*B1
pmull2 t5.8h, t5.16b, bd.16b // H = A2*B pmull2 t5.8h, t5.16b, fold_consts.16b // H = A2*B
pmull2 t7.8h, ad.16b, bd2.16b // G = A*B2 pmull2 t7.8h, ad.16b, bd2.16b // G = A*B2
pmull2 t6.8h, t6.16b, bd.16b // J = A3*B pmull2 t6.8h, t6.16b, fold_consts.16b // J = A3*B
pmull2 t9.8h, ad.16b, bd3.16b // I = A*B3 pmull2 t9.8h, ad.16b, bd3.16b // I = A*B3
pmull2 t3.8h, ad.16b, bd4.16b // K = A*B4 pmull2 t3.8h, ad.16b, bd4.16b // K = A*B4
@ -203,14 +197,14 @@ __pmull_p8_core:
ENDPROC(__pmull_p8_core) ENDPROC(__pmull_p8_core)
.macro __pmull_p8, rq, ad, bd, i .macro __pmull_p8, rq, ad, bd, i
.ifnc \bd, v10 .ifnc \bd, fold_consts
.err .err
.endif .endif
mov ad.16b, \ad\().16b mov ad.16b, \ad\().16b
.ifb \i .ifb \i
pmull \rq\().8h, \ad\().8b, bd.8b // D = A*B pmull \rq\().8h, \ad\().8b, \bd\().8b // D = A*B
.else .else
pmull2 \rq\().8h, \ad\().16b, bd.16b // D = A*B pmull2 \rq\().8h, \ad\().16b, \bd\().16b // D = A*B
.endif .endif
bl .L__pmull_p8_core\i bl .L__pmull_p8_core\i
@ -219,17 +213,19 @@ ENDPROC(__pmull_p8_core)
eor \rq\().16b, \rq\().16b, t6.16b eor \rq\().16b, \rq\().16b, t6.16b
.endm .endm
.macro fold64, p, reg1, reg2 // Fold reg1, reg2 into the next 32 data bytes, storing the result back
ldp q11, q12, [arg2], #0x20 // into reg1, reg2.
.macro fold_32_bytes, p, reg1, reg2
ldp q11, q12, [buf], #0x20
__pmull_\p v8, \reg1, v10, 2 __pmull_\p v8, \reg1, fold_consts, 2
__pmull_\p \reg1, \reg1, v10 __pmull_\p \reg1, \reg1, fold_consts
CPU_LE( rev64 v11.16b, v11.16b ) CPU_LE( rev64 v11.16b, v11.16b )
CPU_LE( rev64 v12.16b, v12.16b ) CPU_LE( rev64 v12.16b, v12.16b )
__pmull_\p v9, \reg2, v10, 2 __pmull_\p v9, \reg2, fold_consts, 2
__pmull_\p \reg2, \reg2, v10 __pmull_\p \reg2, \reg2, fold_consts
CPU_LE( ext v11.16b, v11.16b, v11.16b, #8 ) CPU_LE( ext v11.16b, v11.16b, v11.16b, #8 )
CPU_LE( ext v12.16b, v12.16b, v12.16b, #8 ) CPU_LE( ext v12.16b, v12.16b, v12.16b, #8 )
@ -240,15 +236,16 @@ CPU_LE( ext v12.16b, v12.16b, v12.16b, #8 )
eor \reg2\().16b, \reg2\().16b, v12.16b eor \reg2\().16b, \reg2\().16b, v12.16b
.endm .endm
.macro fold16, p, reg, rk // Fold src_reg into dst_reg, optionally loading the next fold constants
__pmull_\p v8, \reg, v10 .macro fold_16_bytes, p, src_reg, dst_reg, load_next_consts
__pmull_\p \reg, \reg, v10, 2 __pmull_\p v8, \src_reg, fold_consts
.ifnb \rk __pmull_\p \src_reg, \src_reg, fold_consts, 2
ldr_l q10, \rk, x8 .ifnb \load_next_consts
__pmull_pre_\p v10 ld1 {fold_consts.2d}, [fold_consts_ptr], #16
__pmull_pre_\p fold_consts
.endif .endif
eor v7.16b, v7.16b, v8.16b eor \dst_reg\().16b, \dst_reg\().16b, v8.16b
eor v7.16b, v7.16b, \reg\().16b eor \dst_reg\().16b, \dst_reg\().16b, \src_reg\().16b
.endm .endm
.macro __pmull_p64, rd, rn, rm, n .macro __pmull_p64, rd, rn, rm, n
@ -260,40 +257,27 @@ CPU_LE( ext v12.16b, v12.16b, v12.16b, #8 )
.endm .endm
.macro crc_t10dif_pmull, p .macro crc_t10dif_pmull, p
frame_push 3, 128 frame_push 4, 128
mov arg1_low32, w0 mov init_crc, w0
mov arg2, x1 mov buf, x1
mov arg3, x2 mov len, x2
movi vzr.16b, #0 // init zero register
__pmull_init_\p __pmull_init_\p
// adjust the 16-bit initial_crc value, scale it to 32 bits // For sizes less than 256 bytes, we can't fold 128 bytes at a time.
lsl arg1_low32, arg1_low32, #16 cmp len, #256
b.lt .Lless_than_256_bytes_\@
// check if smaller than 256 adr_l fold_consts_ptr, .Lfold_across_128_bytes_consts
cmp arg3, #256
// for sizes less than 128, we can't fold 64B at a time...
b.lt .L_less_than_128_\@
// load the initial crc value
// crc value does not need to be byte-reflected, but it needs
// to be moved to the high part of the register.
// because data will be byte-reflected and will align with
// initial crc at correct place.
movi v10.16b, #0
mov v10.s[3], arg1_low32 // initial crc
// receive the initial 64B data, xor the initial crc value
ldp q0, q1, [arg2]
ldp q2, q3, [arg2, #0x20]
ldp q4, q5, [arg2, #0x40]
ldp q6, q7, [arg2, #0x60]
add arg2, arg2, #0x80
// Load the first 128 data bytes. Byte swapping is necessary to make
// the bit order match the polynomial coefficient order.
ldp q0, q1, [buf]
ldp q2, q3, [buf, #0x20]
ldp q4, q5, [buf, #0x40]
ldp q6, q7, [buf, #0x60]
add buf, buf, #0x80
CPU_LE( rev64 v0.16b, v0.16b ) CPU_LE( rev64 v0.16b, v0.16b )
CPU_LE( rev64 v1.16b, v1.16b ) CPU_LE( rev64 v1.16b, v1.16b )
CPU_LE( rev64 v2.16b, v2.16b ) CPU_LE( rev64 v2.16b, v2.16b )
@ -302,7 +286,6 @@ CPU_LE( rev64 v4.16b, v4.16b )
CPU_LE( rev64 v5.16b, v5.16b ) CPU_LE( rev64 v5.16b, v5.16b )
CPU_LE( rev64 v6.16b, v6.16b ) CPU_LE( rev64 v6.16b, v6.16b )
CPU_LE( rev64 v7.16b, v7.16b ) CPU_LE( rev64 v7.16b, v7.16b )
CPU_LE( ext v0.16b, v0.16b, v0.16b, #8 ) CPU_LE( ext v0.16b, v0.16b, v0.16b, #8 )
CPU_LE( ext v1.16b, v1.16b, v1.16b, #8 ) CPU_LE( ext v1.16b, v1.16b, v1.16b, #8 )
CPU_LE( ext v2.16b, v2.16b, v2.16b, #8 ) CPU_LE( ext v2.16b, v2.16b, v2.16b, #8 )
@ -312,36 +295,29 @@ CPU_LE( ext v5.16b, v5.16b, v5.16b, #8 )
CPU_LE( ext v6.16b, v6.16b, v6.16b, #8 ) CPU_LE( ext v6.16b, v6.16b, v6.16b, #8 )
CPU_LE( ext v7.16b, v7.16b, v7.16b, #8 ) CPU_LE( ext v7.16b, v7.16b, v7.16b, #8 )
// XOR the initial_crc value // XOR the first 16 data *bits* with the initial CRC value.
eor v0.16b, v0.16b, v10.16b movi v8.16b, #0
mov v8.h[7], init_crc
eor v0.16b, v0.16b, v8.16b
ldr_l q10, rk3, x8 // xmm10 has rk3 and rk4 // Load the constants for folding across 128 bytes.
// type of pmull instruction ld1 {fold_consts.2d}, [fold_consts_ptr]
// will determine which constant to use __pmull_pre_\p fold_consts
__pmull_pre_\p v10
// // Subtract 128 for the 128 data bytes just consumed. Subtract another
// we subtract 256 instead of 128 to save one instruction from the loop // 128 to simplify the termination condition of the following loop.
// sub len, len, #256
sub arg3, arg3, #256
// at this section of the code, there is 64*x+y (0<=y<64) bytes of // While >= 128 data bytes remain (not counting v0-v7), fold the 128
// buffer. The _fold_64_B_loop will fold 64B at a time // bytes v0-v7 into them, storing the result back into v0-v7.
// until we have 64+y Bytes of buffer .Lfold_128_bytes_loop_\@:
fold_32_bytes \p, v0, v1
fold_32_bytes \p, v2, v3
fold_32_bytes \p, v4, v5
fold_32_bytes \p, v6, v7
// fold 64B at a time. This section of the code folds 4 vector subs len, len, #128
// registers in parallel b.lt .Lfold_128_bytes_loop_done_\@
.L_fold_64_B_loop_\@:
fold64 \p, v0, v1
fold64 \p, v2, v3
fold64 \p, v4, v5
fold64 \p, v6, v7
subs arg3, arg3, #128
// check if there is another 64B in the buffer to be able to fold
b.lt .L_fold_64_B_end_\@
if_will_cond_yield_neon if_will_cond_yield_neon
stp q0, q1, [sp, #.Lframe_local_offset] stp q0, q1, [sp, #.Lframe_local_offset]
@ -353,228 +329,207 @@ CPU_LE( ext v7.16b, v7.16b, v7.16b, #8 )
ldp q2, q3, [sp, #.Lframe_local_offset + 32] ldp q2, q3, [sp, #.Lframe_local_offset + 32]
ldp q4, q5, [sp, #.Lframe_local_offset + 64] ldp q4, q5, [sp, #.Lframe_local_offset + 64]
ldp q6, q7, [sp, #.Lframe_local_offset + 96] ldp q6, q7, [sp, #.Lframe_local_offset + 96]
ldr_l q10, rk3, x8 ld1 {fold_consts.2d}, [fold_consts_ptr]
movi vzr.16b, #0 // init zero register
__pmull_init_\p __pmull_init_\p
__pmull_pre_\p v10 __pmull_pre_\p fold_consts
endif_yield_neon endif_yield_neon
b .L_fold_64_B_loop_\@ b .Lfold_128_bytes_loop_\@
.L_fold_64_B_end_\@: .Lfold_128_bytes_loop_done_\@:
// at this point, the buffer pointer is pointing at the last y Bytes
// of the buffer the 64B of folded data is in 4 of the vector
// registers: v0, v1, v2, v3
// fold the 8 vector registers to 1 vector register with different // Now fold the 112 bytes in v0-v6 into the 16 bytes in v7.
// constants
ldr_l q10, rk9, x8 // Fold across 64 bytes.
__pmull_pre_\p v10 add fold_consts_ptr, fold_consts_ptr, #16
ld1 {fold_consts.2d}, [fold_consts_ptr], #16
__pmull_pre_\p fold_consts
fold_16_bytes \p, v0, v4
fold_16_bytes \p, v1, v5
fold_16_bytes \p, v2, v6
fold_16_bytes \p, v3, v7, 1
// Fold across 32 bytes.
fold_16_bytes \p, v4, v6
fold_16_bytes \p, v5, v7, 1
// Fold across 16 bytes.
fold_16_bytes \p, v6, v7
fold16 \p, v0, rk11 // Add 128 to get the correct number of data bytes remaining in 0...127
fold16 \p, v1, rk13 // (not counting v7), following the previous extra subtraction by 128.
fold16 \p, v2, rk15 // Then subtract 16 to simplify the termination condition of the
fold16 \p, v3, rk17 // following loop.
fold16 \p, v4, rk19 adds len, len, #(128-16)
fold16 \p, v5, rk1
fold16 \p, v6
// instead of 64, we add 48 to the loop counter to save 1 instruction // While >= 16 data bytes remain (not counting v7), fold the 16 bytes v7
// from the loop instead of a cmp instruction, we use the negative // into them, storing the result back into v7.
// flag with the jl instruction b.lt .Lfold_16_bytes_loop_done_\@
adds arg3, arg3, #(128-16) .Lfold_16_bytes_loop_\@:
b.lt .L_final_reduction_for_128_\@ __pmull_\p v8, v7, fold_consts
__pmull_\p v7, v7, fold_consts, 2
// now we have 16+y bytes left to reduce. 16 Bytes is in register v7
// and the rest is in memory. We can fold 16 bytes at a time if y>=16
// continue folding 16B at a time
.L_16B_reduction_loop_\@:
__pmull_\p v8, v7, v10
__pmull_\p v7, v7, v10, 2
eor v7.16b, v7.16b, v8.16b eor v7.16b, v7.16b, v8.16b
ldr q0, [buf], #16
ldr q0, [arg2], #16
CPU_LE( rev64 v0.16b, v0.16b ) CPU_LE( rev64 v0.16b, v0.16b )
CPU_LE( ext v0.16b, v0.16b, v0.16b, #8 ) CPU_LE( ext v0.16b, v0.16b, v0.16b, #8 )
eor v7.16b, v7.16b, v0.16b eor v7.16b, v7.16b, v0.16b
subs arg3, arg3, #16 subs len, len, #16
b.ge .Lfold_16_bytes_loop_\@
// instead of a cmp instruction, we utilize the flags with the .Lfold_16_bytes_loop_done_\@:
// jge instruction equivalent of: cmp arg3, 16-16 // Add 16 to get the correct number of data bytes remaining in 0...15
// check if there is any more 16B in the buffer to be able to fold // (not counting v7), following the previous extra subtraction by 16.
b.ge .L_16B_reduction_loop_\@ adds len, len, #16
b.eq .Lreduce_final_16_bytes_\@
// now we have 16+z bytes left to reduce, where 0<= z < 16. .Lhandle_partial_segment_\@:
// first, we reduce the data in the xmm7 register // Reduce the last '16 + len' bytes where 1 <= len <= 15 and the first
// 16 bytes are in v7 and the rest are the remaining data in 'buf'. To
// do this without needing a fold constant for each possible 'len',
// redivide the bytes into a first chunk of 'len' bytes and a second
// chunk of 16 bytes, then fold the first chunk into the second.
.L_final_reduction_for_128_\@: // v0 = last 16 original data bytes
// check if any more data to fold. If not, compute the CRC of add buf, buf, len
// the final 128 bits ldr q0, [buf, #-16]
adds arg3, arg3, #16 CPU_LE( rev64 v0.16b, v0.16b )
b.eq .L_128_done_\@ CPU_LE( ext v0.16b, v0.16b, v0.16b, #8 )
// here we are getting data that is less than 16 bytes. // v1 = high order part of second chunk: v7 left-shifted by 'len' bytes.
// since we know that there was data before the pointer, we can adr_l x4, .Lbyteshift_table + 16
// offset the input pointer before the actual point, to receive sub x4, x4, len
// exactly 16 bytes. after that the registers need to be adjusted. ld1 {v2.16b}, [x4]
.L_get_last_two_regs_\@: tbl v1.16b, {v7.16b}, v2.16b
add arg2, arg2, arg3
ldr q1, [arg2, #-16]
CPU_LE( rev64 v1.16b, v1.16b )
CPU_LE( ext v1.16b, v1.16b, v1.16b, #8 )
// get rid of the extra data that was loaded before // v3 = first chunk: v7 right-shifted by '16-len' bytes.
// load the shift constant movi v3.16b, #0x80
adr_l x4, tbl_shf_table + 16 eor v2.16b, v2.16b, v3.16b
sub x4, x4, arg3 tbl v3.16b, {v7.16b}, v2.16b
ld1 {v0.16b}, [x4]
// shift v2 to the left by arg3 bytes // Convert to 8-bit masks: 'len' 0x00 bytes, then '16-len' 0xff bytes.
tbl v2.16b, {v7.16b}, v0.16b sshr v2.16b, v2.16b, #7
// shift v7 to the right by 16-arg3 bytes // v2 = second chunk: 'len' bytes from v0 (low-order bytes),
movi v9.16b, #0x80 // then '16-len' bytes from v1 (high-order bytes).
eor v0.16b, v0.16b, v9.16b bsl v2.16b, v1.16b, v0.16b
tbl v7.16b, {v7.16b}, v0.16b
// blend // Fold the first chunk into the second chunk, storing the result in v7.
sshr v0.16b, v0.16b, #7 // convert to 8-bit mask __pmull_\p v0, v3, fold_consts
bsl v0.16b, v2.16b, v1.16b __pmull_\p v7, v3, fold_consts, 2
// fold 16 Bytes
__pmull_\p v8, v7, v10
__pmull_\p v7, v7, v10, 2
eor v7.16b, v7.16b, v8.16b
eor v7.16b, v7.16b, v0.16b eor v7.16b, v7.16b, v0.16b
eor v7.16b, v7.16b, v2.16b
.L_128_done_\@: .Lreduce_final_16_bytes_\@:
// compute crc of a 128-bit value // Reduce the 128-bit value M(x), stored in v7, to the final 16-bit CRC.
ldr_l q10, rk5, x8 // rk5 and rk6 in xmm10
__pmull_pre_\p v10
// 64b fold movi v2.16b, #0 // init zero register
ext v0.16b, vzr.16b, v7.16b, #8
mov v7.d[0], v7.d[1]
__pmull_\p v7, v7, v10
eor v7.16b, v7.16b, v0.16b
// 32b fold // Load 'x^48 * (x^48 mod G(x))' and 'x^48 * (x^80 mod G(x))'.
ext v0.16b, v7.16b, vzr.16b, #4 ld1 {fold_consts.2d}, [fold_consts_ptr], #16
mov v7.s[3], vzr.s[0] __pmull_pre_\p fold_consts
__pmull_\p v0, v0, v10, 2
eor v7.16b, v7.16b, v0.16b
// barrett reduction // Fold the high 64 bits into the low 64 bits, while also multiplying by
ldr_l q10, rk7, x8 // x^64. This produces a 128-bit value congruent to x^64 * M(x) and
__pmull_pre_\p v10 // whose low 48 bits are 0.
mov v0.d[0], v7.d[1] ext v0.16b, v2.16b, v7.16b, #8
__pmull_\p v7, v7, fold_consts, 2 // high bits * x^48 * (x^80 mod G(x))
eor v0.16b, v0.16b, v7.16b // + low bits * x^64
__pmull_\p v0, v0, v10 // Fold the high 32 bits into the low 96 bits. This produces a 96-bit
ext v0.16b, vzr.16b, v0.16b, #12 // value congruent to x^64 * M(x) and whose low 48 bits are 0.
__pmull_\p v0, v0, v10, 2 ext v1.16b, v0.16b, v2.16b, #12 // extract high 32 bits
ext v0.16b, vzr.16b, v0.16b, #12 mov v0.s[3], v2.s[0] // zero high 32 bits
eor v7.16b, v7.16b, v0.16b __pmull_\p v1, v1, fold_consts // high 32 bits * x^48 * (x^48 mod G(x))
mov w0, v7.s[1] eor v0.16b, v0.16b, v1.16b // + low bits
.L_cleanup_\@: // Load G(x) and floor(x^48 / G(x)).
// scale the result back to 16 bits ld1 {fold_consts.2d}, [fold_consts_ptr]
lsr x0, x0, #16 __pmull_pre_\p fold_consts
// Use Barrett reduction to compute the final CRC value.
__pmull_\p v1, v0, fold_consts, 2 // high 32 bits * floor(x^48 / G(x))
ushr v1.2d, v1.2d, #32 // /= x^32
__pmull_\p v1, v1, fold_consts // *= G(x)
ushr v0.2d, v0.2d, #48
eor v0.16b, v0.16b, v1.16b // + low 16 nonzero bits
// Final CRC value (x^16 * M(x)) mod G(x) is in low 16 bits of v0.
umov w0, v0.h[0]
frame_pop frame_pop
ret ret
.L_less_than_128_\@: .Lless_than_256_bytes_\@:
cbz arg3, .L_cleanup_\@ // Checksumming a buffer of length 16...255 bytes
movi v0.16b, #0 adr_l fold_consts_ptr, .Lfold_across_16_bytes_consts
mov v0.s[3], arg1_low32 // get the initial crc value
ldr q7, [arg2], #0x10 // Load the first 16 data bytes.
ldr q7, [buf], #0x10
CPU_LE( rev64 v7.16b, v7.16b ) CPU_LE( rev64 v7.16b, v7.16b )
CPU_LE( ext v7.16b, v7.16b, v7.16b, #8 ) CPU_LE( ext v7.16b, v7.16b, v7.16b, #8 )
eor v7.16b, v7.16b, v0.16b // xor the initial crc value
cmp arg3, #16 // XOR the first 16 data *bits* with the initial CRC value.
b.eq .L_128_done_\@ // exactly 16 left movi v0.16b, #0
b.lt .L_less_than_16_left_\@ mov v0.h[7], init_crc
eor v7.16b, v7.16b, v0.16b
ldr_l q10, rk1, x8 // rk1 and rk2 in xmm10 // Load the fold-across-16-bytes constants.
__pmull_pre_\p v10 ld1 {fold_consts.2d}, [fold_consts_ptr], #16
__pmull_pre_\p fold_consts
// update the counter. subtract 32 instead of 16 to save one cmp len, #16
// instruction from the loop b.eq .Lreduce_final_16_bytes_\@ // len == 16
subs arg3, arg3, #32 subs len, len, #32
b.ge .L_16B_reduction_loop_\@ b.ge .Lfold_16_bytes_loop_\@ // 32 <= len <= 255
add len, len, #16
add arg3, arg3, #16 b .Lhandle_partial_segment_\@ // 17 <= len <= 31
b .L_get_last_two_regs_\@
.L_less_than_16_left_\@:
// shl r9, 4
adr_l x0, tbl_shf_table + 16
sub x0, x0, arg3
ld1 {v0.16b}, [x0]
movi v9.16b, #0x80
eor v0.16b, v0.16b, v9.16b
tbl v7.16b, {v7.16b}, v0.16b
b .L_128_done_\@
.endm .endm
//
// u16 crc_t10dif_pmull_p8(u16 init_crc, const u8 *buf, size_t len);
//
// Assumes len >= 16.
//
ENTRY(crc_t10dif_pmull_p8) ENTRY(crc_t10dif_pmull_p8)
crc_t10dif_pmull p8 crc_t10dif_pmull p8
ENDPROC(crc_t10dif_pmull_p8) ENDPROC(crc_t10dif_pmull_p8)
.align 5 .align 5
//
// u16 crc_t10dif_pmull_p64(u16 init_crc, const u8 *buf, size_t len);
//
// Assumes len >= 16.
//
ENTRY(crc_t10dif_pmull_p64) ENTRY(crc_t10dif_pmull_p64)
crc_t10dif_pmull p64 crc_t10dif_pmull p64
ENDPROC(crc_t10dif_pmull_p64) ENDPROC(crc_t10dif_pmull_p64)
// precomputed constants
// these constants are precomputed from the poly:
// 0x8bb70000 (0x8bb7 scaled to 32 bits)
.section ".rodata", "a" .section ".rodata", "a"
.align 4 .align 4
// Q = 0x18BB70000
// rk1 = 2^(32*3) mod Q << 32
// rk2 = 2^(32*5) mod Q << 32
// rk3 = 2^(32*15) mod Q << 32
// rk4 = 2^(32*17) mod Q << 32
// rk5 = 2^(32*3) mod Q << 32
// rk6 = 2^(32*2) mod Q << 32
// rk7 = floor(2^64/Q)
// rk8 = Q
rk1: .octa 0x06df0000000000002d56000000000000 // Fold constants precomputed from the polynomial 0x18bb7
rk3: .octa 0x7cf50000000000009d9d000000000000 // G(x) = x^16 + x^15 + x^11 + x^9 + x^8 + x^7 + x^5 + x^4 + x^2 + x^1 + x^0
rk5: .octa 0x13680000000000002d56000000000000 .Lfold_across_128_bytes_consts:
rk7: .octa 0x000000018bb7000000000001f65a57f8 .quad 0x0000000000006123 // x^(8*128) mod G(x)
rk9: .octa 0xbfd6000000000000ceae000000000000 .quad 0x0000000000002295 // x^(8*128+64) mod G(x)
rk11: .octa 0x713c0000000000001e16000000000000 // .Lfold_across_64_bytes_consts:
rk13: .octa 0x80a6000000000000f7f9000000000000 .quad 0x0000000000001069 // x^(4*128) mod G(x)
rk15: .octa 0xe658000000000000044c000000000000 .quad 0x000000000000dd31 // x^(4*128+64) mod G(x)
rk17: .octa 0xa497000000000000ad18000000000000 // .Lfold_across_32_bytes_consts:
rk19: .octa 0xe7b50000000000006ee3000000000000 .quad 0x000000000000857d // x^(2*128) mod G(x)
.quad 0x0000000000007acc // x^(2*128+64) mod G(x)
tbl_shf_table: .Lfold_across_16_bytes_consts:
// use these values for shift constants for the tbl/tbx instruction .quad 0x000000000000a010 // x^(1*128) mod G(x)
// different alignments result in values as shown: .quad 0x0000000000001faa // x^(1*128+64) mod G(x)
// DDQ 0x008f8e8d8c8b8a898887868584838281 # shl 15 (16-1) / shr1 // .Lfinal_fold_consts:
// DDQ 0x01008f8e8d8c8b8a8988878685848382 # shl 14 (16-3) / shr2 .quad 0x1368000000000000 // x^48 * (x^48 mod G(x))
// DDQ 0x0201008f8e8d8c8b8a89888786858483 # shl 13 (16-4) / shr3 .quad 0x2d56000000000000 // x^48 * (x^80 mod G(x))
// DDQ 0x030201008f8e8d8c8b8a898887868584 # shl 12 (16-4) / shr4 // .Lbarrett_reduction_consts:
// DDQ 0x04030201008f8e8d8c8b8a8988878685 # shl 11 (16-5) / shr5 .quad 0x0000000000018bb7 // G(x)
// DDQ 0x0504030201008f8e8d8c8b8a89888786 # shl 10 (16-6) / shr6 .quad 0x00000001f65a57f8 // floor(x^48 / G(x))
// DDQ 0x060504030201008f8e8d8c8b8a898887 # shl 9 (16-7) / shr7
// DDQ 0x07060504030201008f8e8d8c8b8a8988 # shl 8 (16-8) / shr8
// DDQ 0x0807060504030201008f8e8d8c8b8a89 # shl 7 (16-9) / shr9
// DDQ 0x090807060504030201008f8e8d8c8b8a # shl 6 (16-10) / shr10
// DDQ 0x0a090807060504030201008f8e8d8c8b # shl 5 (16-11) / shr11
// DDQ 0x0b0a090807060504030201008f8e8d8c # shl 4 (16-12) / shr12
// DDQ 0x0c0b0a090807060504030201008f8e8d # shl 3 (16-13) / shr13
// DDQ 0x0d0c0b0a090807060504030201008f8e # shl 2 (16-14) / shr14
// DDQ 0x0e0d0c0b0a090807060504030201008f # shl 1 (16-15) / shr15
// For 1 <= len <= 15, the 16-byte vector beginning at &byteshift_table[16 -
// len] is the index vector to shift left by 'len' bytes, and is also {0x80,
// ..., 0x80} XOR the index vector to shift right by '16 - len' bytes.
.Lbyteshift_table:
.byte 0x0, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87 .byte 0x0, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87
.byte 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f .byte 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f
.byte 0x0, 0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7 .byte 0x0, 0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7

77
arch/arm64/crypto/crct10dif-ce-glue.c
View File

@ -22,10 +22,8 @@
#define CRC_T10DIF_PMULL_CHUNK_SIZE 16U #define CRC_T10DIF_PMULL_CHUNK_SIZE 16U
asmlinkage u16 crc_t10dif_pmull_p64(u16 init_crc, const u8 buf[], u64 len); asmlinkage u16 crc_t10dif_pmull_p8(u16 init_crc, const u8 *buf, size_t len);
asmlinkage u16 crc_t10dif_pmull_p8(u16 init_crc, const u8 buf[], u64 len); asmlinkage u16 crc_t10dif_pmull_p64(u16 init_crc, const u8 *buf, size_t len);
static u16 (*crc_t10dif_pmull)(u16 init_crc, const u8 buf[], u64 len);
static int crct10dif_init(struct shash_desc *desc) static int crct10dif_init(struct shash_desc *desc)
{ {
@ -35,30 +33,33 @@ static int crct10dif_init(struct shash_desc *desc)
return 0; return 0;
} }
static int crct10dif_update(struct shash_desc *desc, const u8 *data, static int crct10dif_update_pmull_p8(struct shash_desc *desc, const u8 *data,
unsigned int length) unsigned int length)
{ {
u16 *crc = shash_desc_ctx(desc); u16 *crc = shash_desc_ctx(desc);
unsigned int l;
if (unlikely((u64)data % CRC_T10DIF_PMULL_CHUNK_SIZE)) { if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) {
l = min_t(u32, length, CRC_T10DIF_PMULL_CHUNK_SIZE - kernel_neon_begin();
((u64)data % CRC_T10DIF_PMULL_CHUNK_SIZE)); *crc = crc_t10dif_pmull_p8(*crc, data, length);
kernel_neon_end();
*crc = crc_t10dif_generic(*crc, data, l); } else {
*crc = crc_t10dif_generic(*crc, data, length);
length -= l;
data += l;
} }
if (length > 0) { return 0;
if (may_use_simd()) { }
kernel_neon_begin();
*crc = crc_t10dif_pmull(*crc, data, length); static int crct10dif_update_pmull_p64(struct shash_desc *desc, const u8 *data,
kernel_neon_end(); unsigned int length)
} else { {
*crc = crc_t10dif_generic(*crc, data, length); u16 *crc = shash_desc_ctx(desc);
}
if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE && may_use_simd()) {
kernel_neon_begin();
*crc = crc_t10dif_pmull_p64(*crc, data, length);
kernel_neon_end();
} else {
*crc = crc_t10dif_generic(*crc, data, length);
} }
return 0; return 0;
@ -72,10 +73,22 @@ static int crct10dif_final(struct shash_desc *desc, u8 *out)
return 0; return 0;
} }
static struct shash_alg crc_t10dif_alg = { static struct shash_alg crc_t10dif_alg[] = {{
.digestsize = CRC_T10DIF_DIGEST_SIZE, .digestsize = CRC_T10DIF_DIGEST_SIZE,
.init = crct10dif_init, .init = crct10dif_init,
.update = crct10dif_update, .update = crct10dif_update_pmull_p8,
.final = crct10dif_final,
.descsize = CRC_T10DIF_DIGEST_SIZE,
.base.cra_name = "crct10dif",
.base.cra_driver_name = "crct10dif-arm64-neon",
.base.cra_priority = 100,
.base.cra_blocksize = CRC_T10DIF_BLOCK_SIZE,
.base.cra_module = THIS_MODULE,
}, {
.digestsize = CRC_T10DIF_DIGEST_SIZE,
.init = crct10dif_init,
.update = crct10dif_update_pmull_p64,
.final = crct10dif_final, .final = crct10dif_final,
.descsize = CRC_T10DIF_DIGEST_SIZE, .descsize = CRC_T10DIF_DIGEST_SIZE,
@ -84,21 +97,25 @@ static struct shash_alg crc_t10dif_alg = {
.base.cra_priority = 200, .base.cra_priority = 200,
.base.cra_blocksize = CRC_T10DIF_BLOCK_SIZE, .base.cra_blocksize = CRC_T10DIF_BLOCK_SIZE,
.base.cra_module = THIS_MODULE, .base.cra_module = THIS_MODULE,
}; }};
static int __init crc_t10dif_mod_init(void) static int __init crc_t10dif_mod_init(void)
{ {
if (elf_hwcap & HWCAP_PMULL) if (elf_hwcap & HWCAP_PMULL)
crc_t10dif_pmull = crc_t10dif_pmull_p64; return crypto_register_shashes(crc_t10dif_alg,
ARRAY_SIZE(crc_t10dif_alg));
else else
crc_t10dif_pmull = crc_t10dif_pmull_p8; /* only register the first array element */
return crypto_register_shash(crc_t10dif_alg);
return crypto_register_shash(&crc_t10dif_alg);
} }
static void __exit crc_t10dif_mod_exit(void) static void __exit crc_t10dif_mod_exit(void)
{ {
crypto_unregister_shash(&crc_t10dif_alg); if (elf_hwcap & HWCAP_PMULL)
crypto_unregister_shashes(crc_t10dif_alg,
ARRAY_SIZE(crc_t10dif_alg));
else
crypto_unregister_shash(crc_t10dif_alg);
} }
module_cpu_feature_match(ASIMD, crc_t10dif_mod_init); module_cpu_feature_match(ASIMD, crc_t10dif_mod_init);

118
arch/arm64/crypto/ghash-ce-glue.c
View File

@ -60,10 +60,6 @@ asmlinkage void pmull_ghash_update_p8(int blocks, u64 dg[], const char *src,
struct ghash_key const *k, struct ghash_key const *k,
const char *head); const char *head);
static void (*pmull_ghash_update)(int blocks, u64 dg[], const char *src,
struct ghash_key const *k,
const char *head);
asmlinkage void pmull_gcm_encrypt(int blocks, u64 dg[], u8 dst[], asmlinkage void pmull_gcm_encrypt(int blocks, u64 dg[], u8 dst[],
const u8 src[], struct ghash_key const *k, const u8 src[], struct ghash_key const *k,
u8 ctr[], u32 const rk[], int rounds, u8 ctr[], u32 const rk[], int rounds,
@ -87,11 +83,15 @@ static int ghash_init(struct shash_desc *desc)
} }
static void ghash_do_update(int blocks, u64 dg[], const char *src, static void ghash_do_update(int blocks, u64 dg[], const char *src,
struct ghash_key *key, const char *head) struct ghash_key *key, const char *head,
void (*simd_update)(int blocks, u64 dg[],
const char *src,
struct ghash_key const *k,
const char *head))
{ {
if (likely(may_use_simd())) { if (likely(may_use_simd())) {
kernel_neon_begin(); kernel_neon_begin();
pmull_ghash_update(blocks, dg, src, key, head); simd_update(blocks, dg, src, key, head);
kernel_neon_end(); kernel_neon_end();
} else { } else {
be128 dst = { cpu_to_be64(dg[1]), cpu_to_be64(dg[0]) }; be128 dst = { cpu_to_be64(dg[1]), cpu_to_be64(dg[0]) };
@ -119,8 +119,12 @@ static void ghash_do_update(int blocks, u64 dg[], const char *src,
/* avoid hogging the CPU for too long */ /* avoid hogging the CPU for too long */
#define MAX_BLOCKS (SZ_64K / GHASH_BLOCK_SIZE) #define MAX_BLOCKS (SZ_64K / GHASH_BLOCK_SIZE)
static int ghash_update(struct shash_desc *desc, const u8 *src, static int __ghash_update(struct shash_desc *desc, const u8 *src,
unsigned int len) unsigned int len,
void (*simd_update)(int blocks, u64 dg[],
const char *src,
struct ghash_key const *k,
const char *head))
{ {
struct ghash_desc_ctx *ctx = shash_desc_ctx(desc); struct ghash_desc_ctx *ctx = shash_desc_ctx(desc);
unsigned int partial = ctx->count % GHASH_BLOCK_SIZE; unsigned int partial = ctx->count % GHASH_BLOCK_SIZE;
@ -146,7 +150,8 @@ static int ghash_update(struct shash_desc *desc, const u8 *src,
int chunk = min(blocks, MAX_BLOCKS); int chunk = min(blocks, MAX_BLOCKS);
ghash_do_update(chunk, ctx->digest, src, key, ghash_do_update(chunk, ctx->digest, src, key,
partial ? ctx->buf : NULL); partial ? ctx->buf : NULL,
simd_update);
blocks -= chunk; blocks -= chunk;
src += chunk * GHASH_BLOCK_SIZE; src += chunk * GHASH_BLOCK_SIZE;
@ -158,7 +163,19 @@ static int ghash_update(struct shash_desc *desc, const u8 *src,
return 0; return 0;
} }
static int ghash_final(struct shash_desc *desc, u8 *dst) static int ghash_update_p8(struct shash_desc *desc, const u8 *src,
unsigned int len)
{
return __ghash_update(desc, src, len, pmull_ghash_update_p8);
}
static int ghash_update_p64(struct shash_desc *desc, const u8 *src,
unsigned int len)
{
return __ghash_update(desc, src, len, pmull_ghash_update_p64);
}
static int ghash_final_p8(struct shash_desc *desc, u8 *dst)
{ {
struct ghash_desc_ctx *ctx = shash_desc_ctx(desc); struct ghash_desc_ctx *ctx = shash_desc_ctx(desc);
unsigned int partial = ctx->count % GHASH_BLOCK_SIZE; unsigned int partial = ctx->count % GHASH_BLOCK_SIZE;
@ -168,7 +185,28 @@ static int ghash_final(struct shash_desc *desc, u8 *dst)
memset(ctx->buf + partial, 0, GHASH_BLOCK_SIZE - partial); memset(ctx->buf + partial, 0, GHASH_BLOCK_SIZE - partial);
ghash_do_update(1, ctx->digest, ctx->buf, key, NULL); ghash_do_update(1, ctx->digest, ctx->buf, key, NULL,
pmull_ghash_update_p8);
}
put_unaligned_be64(ctx->digest[1], dst);
put_unaligned_be64(ctx->digest[0], dst + 8);
*ctx = (struct ghash_desc_ctx){};
return 0;
}
static int ghash_final_p64(struct shash_desc *desc, u8 *dst)
{
struct ghash_desc_ctx *ctx = shash_desc_ctx(desc);
unsigned int partial = ctx->count % GHASH_BLOCK_SIZE;
if (partial) {
struct ghash_key *key = crypto_shash_ctx(desc->tfm);
memset(ctx->buf + partial, 0, GHASH_BLOCK_SIZE - partial);
ghash_do_update(1, ctx->digest, ctx->buf, key, NULL,
pmull_ghash_update_p64);
} }
put_unaligned_be64(ctx->digest[1], dst); put_unaligned_be64(ctx->digest[1], dst);
put_unaligned_be64(ctx->digest[0], dst + 8); put_unaligned_be64(ctx->digest[0], dst + 8);
@ -224,7 +262,21 @@ static int ghash_setkey(struct crypto_shash *tfm,
return __ghash_setkey(key, inkey, keylen); return __ghash_setkey(key, inkey, keylen);
} }
static struct shash_alg ghash_alg = { static struct shash_alg ghash_alg[] = {{
.base.cra_name = "ghash",
.base.cra_driver_name = "ghash-neon",
.base.cra_priority = 100,
.base.cra_blocksize = GHASH_BLOCK_SIZE,
.base.cra_ctxsize = sizeof(struct ghash_key),
.base.cra_module = THIS_MODULE,
.digestsize = GHASH_DIGEST_SIZE,
.init = ghash_init,
.update = ghash_update_p8,
.final = ghash_final_p8,
.setkey = ghash_setkey,
.descsize = sizeof(struct ghash_desc_ctx),
}, {
.base.cra_name = "ghash", .base.cra_name = "ghash",
.base.cra_driver_name = "ghash-ce", .base.cra_driver_name = "ghash-ce",
.base.cra_priority = 200, .base.cra_priority = 200,
@ -234,11 +286,11 @@ static struct shash_alg ghash_alg = {
.digestsize = GHASH_DIGEST_SIZE, .digestsize = GHASH_DIGEST_SIZE,
.init = ghash_init, .init = ghash_init,
.update = ghash_update, .update = ghash_update_p64,
.final = ghash_final, .final = ghash_final_p64,
.setkey = ghash_setkey, .setkey = ghash_setkey,
.descsize = sizeof(struct ghash_desc_ctx), .descsize = sizeof(struct ghash_desc_ctx),
}; }};
static int num_rounds(struct crypto_aes_ctx *ctx) static int num_rounds(struct crypto_aes_ctx *ctx)
{ {
@ -301,7 +353,8 @@ static void gcm_update_mac(u64 dg[], const u8 *src, int count, u8 buf[],
int blocks = count / GHASH_BLOCK_SIZE; int blocks = count / GHASH_BLOCK_SIZE;
ghash_do_update(blocks, dg, src, &ctx->ghash_key, ghash_do_update(blocks, dg, src, &ctx->ghash_key,
*buf_count ? buf : NULL); *buf_count ? buf : NULL,
pmull_ghash_update_p64);
src += blocks * GHASH_BLOCK_SIZE; src += blocks * GHASH_BLOCK_SIZE;
count %= GHASH_BLOCK_SIZE; count %= GHASH_BLOCK_SIZE;
@ -345,7 +398,8 @@ static void gcm_calculate_auth_mac(struct aead_request *req, u64 dg[])
if (buf_count) { if (buf_count) {
memset(&buf[buf_count], 0, GHASH_BLOCK_SIZE - buf_count); memset(&buf[buf_count], 0, GHASH_BLOCK_SIZE - buf_count);
ghash_do_update(1, dg, buf, &ctx->ghash_key, NULL); ghash_do_update(1, dg, buf, &ctx->ghash_key, NULL,
pmull_ghash_update_p64);
} }
} }
@ -358,7 +412,8 @@ static void gcm_final(struct aead_request *req, struct gcm_aes_ctx *ctx,
lengths.a = cpu_to_be64(req->assoclen * 8); lengths.a = cpu_to_be64(req->assoclen * 8);
lengths.b = cpu_to_be64(cryptlen * 8); lengths.b = cpu_to_be64(cryptlen * 8);
ghash_do_update(1, dg, (void *)&lengths, &ctx->ghash_key, NULL); ghash_do_update(1, dg, (void *)&lengths, &ctx->ghash_key, NULL,
pmull_ghash_update_p64);
put_unaligned_be64(dg[1], mac); put_unaligned_be64(dg[1], mac);
put_unaligned_be64(dg[0], mac + 8); put_unaligned_be64(dg[0], mac + 8);
@ -434,7 +489,7 @@ static int gcm_encrypt(struct aead_request *req)
ghash_do_update(walk.nbytes / AES_BLOCK_SIZE, dg, ghash_do_update(walk.nbytes / AES_BLOCK_SIZE, dg,
walk.dst.virt.addr, &ctx->ghash_key, walk.dst.virt.addr, &ctx->ghash_key,
NULL); NULL, pmull_ghash_update_p64);
err = skcipher_walk_done(&walk, err = skcipher_walk_done(&walk,
walk.nbytes % (2 * AES_BLOCK_SIZE)); walk.nbytes % (2 * AES_BLOCK_SIZE));
@ -469,7 +524,8 @@ static int gcm_encrypt(struct aead_request *req)
memcpy(buf, dst, nbytes); memcpy(buf, dst, nbytes);
memset(buf + nbytes, 0, GHASH_BLOCK_SIZE - nbytes); memset(buf + nbytes, 0, GHASH_BLOCK_SIZE - nbytes);
ghash_do_update(!!nbytes, dg, buf, &ctx->ghash_key, head); ghash_do_update(!!nbytes, dg, buf, &ctx->ghash_key, head,
pmull_ghash_update_p64);
err = skcipher_walk_done(&walk, 0); err = skcipher_walk_done(&walk, 0);
} }
@ -558,7 +614,8 @@ static int gcm_decrypt(struct aead_request *req)
u8 *src = walk.src.virt.addr; u8 *src = walk.src.virt.addr;
ghash_do_update(blocks, dg, walk.src.virt.addr, ghash_do_update(blocks, dg, walk.src.virt.addr,
&ctx->ghash_key, NULL); &ctx->ghash_key, NULL,
pmull_ghash_update_p64);
do { do {
__aes_arm64_encrypt(ctx->aes_key.key_enc, __aes_arm64_encrypt(ctx->aes_key.key_enc,
@ -602,7 +659,8 @@ static int gcm_decrypt(struct aead_request *req)
memcpy(buf, src, nbytes); memcpy(buf, src, nbytes);
memset(buf + nbytes, 0, GHASH_BLOCK_SIZE - nbytes); memset(buf + nbytes, 0, GHASH_BLOCK_SIZE - nbytes);
ghash_do_update(!!nbytes, dg, buf, &ctx->ghash_key, head); ghash_do_update(!!nbytes, dg, buf, &ctx->ghash_key, head,
pmull_ghash_update_p64);
crypto_xor_cpy(walk.dst.virt.addr, walk.src.virt.addr, iv, crypto_xor_cpy(walk.dst.virt.addr, walk.src.virt.addr, iv,
walk.nbytes); walk.nbytes);
@ -650,26 +708,30 @@ static int __init ghash_ce_mod_init(void)
return -ENODEV; return -ENODEV;
if (elf_hwcap & HWCAP_PMULL) if (elf_hwcap & HWCAP_PMULL)
pmull_ghash_update = pmull_ghash_update_p64; ret = crypto_register_shashes(ghash_alg,
ARRAY_SIZE(ghash_alg));
else else
pmull_ghash_update = pmull_ghash_update_p8; /* only register the first array element */
ret = crypto_register_shash(ghash_alg);
ret = crypto_register_shash(&ghash_alg);
if (ret) if (ret)
return ret; return ret;
if (elf_hwcap & HWCAP_PMULL) { if (elf_hwcap & HWCAP_PMULL) {
ret = crypto_register_aead(&gcm_aes_alg); ret = crypto_register_aead(&gcm_aes_alg);
if (ret) if (ret)
crypto_unregister_shash(&ghash_alg); crypto_unregister_shashes(ghash_alg,
ARRAY_SIZE(ghash_alg));
} }
return ret; return ret;
} }
static void __exit ghash_ce_mod_exit(void) static void __exit ghash_ce_mod_exit(void)
{ {
crypto_unregister_shash(&ghash_alg); if (elf_hwcap & HWCAP_PMULL)
crypto_unregister_shashes(ghash_alg, ARRAY_SIZE(ghash_alg));
else
crypto_unregister_shash(ghash_alg);
crypto_unregister_aead(&gcm_aes_alg); crypto_unregister_aead(&gcm_aes_alg);
} }

4
arch/s390/crypto/des_s390.c
View File

@ -38,7 +38,7 @@ static int des_setkey(struct crypto_tfm *tfm, const u8 *key,
/* check for weak keys */ /* check for weak keys */
if (!des_ekey(tmp, key) && if (!des_ekey(tmp, key) &&
(tfm->crt_flags & CRYPTO_TFM_REQ_WEAK_KEY)) { (tfm->crt_flags & CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) {
tfm->crt_flags |= CRYPTO_TFM_RES_WEAK_KEY; tfm->crt_flags |= CRYPTO_TFM_RES_WEAK_KEY;
return -EINVAL; return -EINVAL;
} }
@ -228,7 +228,7 @@ static int des3_setkey(struct crypto_tfm *tfm, const u8 *key,
if (!(crypto_memneq(key, &key[DES_KEY_SIZE], DES_KEY_SIZE) && if (!(crypto_memneq(key, &key[DES_KEY_SIZE], DES_KEY_SIZE) &&
crypto_memneq(&key[DES_KEY_SIZE], &key[DES_KEY_SIZE * 2], crypto_memneq(&key[DES_KEY_SIZE], &key[DES_KEY_SIZE * 2],
DES_KEY_SIZE)) && DES_KEY_SIZE)) &&
(tfm->crt_flags & CRYPTO_TFM_REQ_WEAK_KEY)) { (tfm->crt_flags & CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) {
tfm->crt_flags |= CRYPTO_TFM_RES_WEAK_KEY; tfm->crt_flags |= CRYPTO_TFM_RES_WEAK_KEY;
return -EINVAL; return -EINVAL;
} }

4
arch/sparc/crypto/des_glue.c
View File

@ -53,7 +53,7 @@ static int des_set_key(struct crypto_tfm *tfm, const u8 *key,
* weak key detection code. * weak key detection code.
*/ */
ret = des_ekey(tmp, key); ret = des_ekey(tmp, key);
if (unlikely(ret == 0) && (*flags & CRYPTO_TFM_REQ_WEAK_KEY)) { if (unlikely(ret == 0) && (*flags & CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) {
*flags |= CRYPTO_TFM_RES_WEAK_KEY; *flags |= CRYPTO_TFM_RES_WEAK_KEY;
return -EINVAL; return -EINVAL;
} }
@ -209,7 +209,7 @@ static int des3_ede_set_key(struct crypto_tfm *tfm, const u8 *key,
if (unlikely(!((K[0] ^ K[2]) | (K[1] ^ K[3])) || if (unlikely(!((K[0] ^ K[2]) | (K[1] ^ K[3])) ||
!((K[2] ^ K[4]) | (K[3] ^ K[5]))) && !((K[2] ^ K[4]) | (K[3] ^ K[5]))) &&
(*flags & CRYPTO_TFM_REQ_WEAK_KEY)) { (*flags & CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) {
*flags |= CRYPTO_TFM_RES_WEAK_KEY; *flags |= CRYPTO_TFM_RES_WEAK_KEY;
return -EINVAL; return -EINVAL;
} }

38
arch/x86/crypto/aegis128-aesni-glue.c
View File

@ -119,31 +119,20 @@ static void crypto_aegis128_aesni_process_ad(
} }
static void crypto_aegis128_aesni_process_crypt( static void crypto_aegis128_aesni_process_crypt(
struct aegis_state *state, struct aead_request *req, struct aegis_state *state, struct skcipher_walk *walk,
const struct aegis_crypt_ops *ops) const struct aegis_crypt_ops *ops)
{ {
struct skcipher_walk walk; while (walk->nbytes >= AEGIS128_BLOCK_SIZE) {
u8 *src, *dst; ops->crypt_blocks(state,
unsigned int chunksize, base; round_down(walk->nbytes, AEGIS128_BLOCK_SIZE),
walk->src.virt.addr, walk->dst.virt.addr);
skcipher_walk_done(walk, walk->nbytes % AEGIS128_BLOCK_SIZE);
}
ops->skcipher_walk_init(&walk, req, false); if (walk->nbytes) {
ops->crypt_tail(state, walk->nbytes, walk->src.virt.addr,
while (walk.nbytes) { walk->dst.virt.addr);
src = walk.src.virt.addr; skcipher_walk_done(walk, 0);
dst = walk.dst.virt.addr;
chunksize = walk.nbytes;
ops->crypt_blocks(state, chunksize, src, dst);
base = chunksize & ~(AEGIS128_BLOCK_SIZE - 1);
src += base;
dst += base;
chunksize &= AEGIS128_BLOCK_SIZE - 1;
if (chunksize > 0)
ops->crypt_tail(state, chunksize, src, dst);
skcipher_walk_done(&walk, 0);
} }
} }
@ -186,13 +175,16 @@ static void crypto_aegis128_aesni_crypt(struct aead_request *req,
{ {
struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct aegis_ctx *ctx = crypto_aegis128_aesni_ctx(tfm); struct aegis_ctx *ctx = crypto_aegis128_aesni_ctx(tfm);
struct skcipher_walk walk;
struct aegis_state state; struct aegis_state state;
ops->skcipher_walk_init(&walk, req, true);
kernel_fpu_begin(); kernel_fpu_begin();
crypto_aegis128_aesni_init(&state, ctx->key.bytes, req->iv); crypto_aegis128_aesni_init(&state, ctx->key.bytes, req->iv);
crypto_aegis128_aesni_process_ad(&state, req->src, req->assoclen); crypto_aegis128_aesni_process_ad(&state, req->src, req->assoclen);
crypto_aegis128_aesni_process_crypt(&state, req, ops); crypto_aegis128_aesni_process_crypt(&state, &walk, ops);
crypto_aegis128_aesni_final(&state, tag_xor, req->assoclen, cryptlen); crypto_aegis128_aesni_final(&state, tag_xor, req->assoclen, cryptlen);
kernel_fpu_end(); kernel_fpu_end();

38
arch/x86/crypto/aegis128l-aesni-glue.c
View File

@ -119,31 +119,20 @@ static void crypto_aegis128l_aesni_process_ad(
} }
static void crypto_aegis128l_aesni_process_crypt( static void crypto_aegis128l_aesni_process_crypt(
struct aegis_state *state, struct aead_request *req, struct aegis_state *state, struct skcipher_walk *walk,
const struct aegis_crypt_ops *ops) const struct aegis_crypt_ops *ops)
{ {
struct skcipher_walk walk; while (walk->nbytes >= AEGIS128L_BLOCK_SIZE) {
u8 *src, *dst; ops->crypt_blocks(state, round_down(walk->nbytes,
unsigned int chunksize, base; AEGIS128L_BLOCK_SIZE),
walk->src.virt.addr, walk->dst.virt.addr);
skcipher_walk_done(walk, walk->nbytes % AEGIS128L_BLOCK_SIZE);
}
ops->skcipher_walk_init(&walk, req, false); if (walk->nbytes) {
ops->crypt_tail(state, walk->nbytes, walk->src.virt.addr,
while (walk.nbytes) { walk->dst.virt.addr);
src = walk.src.virt.addr; skcipher_walk_done(walk, 0);
dst = walk.dst.virt.addr;
chunksize = walk.nbytes;
ops->crypt_blocks(state, chunksize, src, dst);
base = chunksize & ~(AEGIS128L_BLOCK_SIZE - 1);
src += base;
dst += base;
chunksize &= AEGIS128L_BLOCK_SIZE - 1;
if (chunksize > 0)
ops->crypt_tail(state, chunksize, src, dst);
skcipher_walk_done(&walk, 0);
} }
} }
@ -186,13 +175,16 @@ static void crypto_aegis128l_aesni_crypt(struct aead_request *req,
{ {
struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct aegis_ctx *ctx = crypto_aegis128l_aesni_ctx(tfm); struct aegis_ctx *ctx = crypto_aegis128l_aesni_ctx(tfm);
struct skcipher_walk walk;
struct aegis_state state; struct aegis_state state;
ops->skcipher_walk_init(&walk, req, true);
kernel_fpu_begin(); kernel_fpu_begin();
crypto_aegis128l_aesni_init(&state, ctx->key.bytes, req->iv); crypto_aegis128l_aesni_init(&state, ctx->key.bytes, req->iv);
crypto_aegis128l_aesni_process_ad(&state, req->src, req->assoclen); crypto_aegis128l_aesni_process_ad(&state, req->src, req->assoclen);
crypto_aegis128l_aesni_process_crypt(&state, req, ops); crypto_aegis128l_aesni_process_crypt(&state, &walk, ops);
crypto_aegis128l_aesni_final(&state, tag_xor, req->assoclen, cryptlen); crypto_aegis128l_aesni_final(&state, tag_xor, req->assoclen, cryptlen);
kernel_fpu_end(); kernel_fpu_end();

38
arch/x86/crypto/aegis256-aesni-glue.c
View File

@ -119,31 +119,20 @@ static void crypto_aegis256_aesni_process_ad(
} }
static void crypto_aegis256_aesni_process_crypt( static void crypto_aegis256_aesni_process_crypt(
struct aegis_state *state, struct aead_request *req, struct aegis_state *state, struct skcipher_walk *walk,
const struct aegis_crypt_ops *ops) const struct aegis_crypt_ops *ops)
{ {
struct skcipher_walk walk; while (walk->nbytes >= AEGIS256_BLOCK_SIZE) {
u8 *src, *dst; ops->crypt_blocks(state,
unsigned int chunksize, base; round_down(walk->nbytes, AEGIS256_BLOCK_SIZE),
walk->src.virt.addr, walk->dst.virt.addr);
skcipher_walk_done(walk, walk->nbytes % AEGIS256_BLOCK_SIZE);
}
ops->skcipher_walk_init(&walk, req, false); if (walk->nbytes) {
ops->crypt_tail(state, walk->nbytes, walk->src.virt.addr,
while (walk.nbytes) { walk->dst.virt.addr);
src = walk.src.virt.addr; skcipher_walk_done(walk, 0);
dst = walk.dst.virt.addr;
chunksize = walk.nbytes;
ops->crypt_blocks(state, chunksize, src, dst);
base = chunksize & ~(AEGIS256_BLOCK_SIZE - 1);
src += base;
dst += base;
chunksize &= AEGIS256_BLOCK_SIZE - 1;
if (chunksize > 0)
ops->crypt_tail(state, chunksize, src, dst);
skcipher_walk_done(&walk, 0);
} }
} }
@ -186,13 +175,16 @@ static void crypto_aegis256_aesni_crypt(struct aead_request *req,
{ {
struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct aegis_ctx *ctx = crypto_aegis256_aesni_ctx(tfm); struct aegis_ctx *ctx = crypto_aegis256_aesni_ctx(tfm);
struct skcipher_walk walk;
struct aegis_state state; struct aegis_state state;
ops->skcipher_walk_init(&walk, req, true);
kernel_fpu_begin(); kernel_fpu_begin();
crypto_aegis256_aesni_init(&state, ctx->key, req->iv); crypto_aegis256_aesni_init(&state, ctx->key, req->iv);
crypto_aegis256_aesni_process_ad(&state, req->src, req->assoclen); crypto_aegis256_aesni_process_ad(&state, req->src, req->assoclen);
crypto_aegis256_aesni_process_crypt(&state, req, ops); crypto_aegis256_aesni_process_crypt(&state, &walk, ops);
crypto_aegis256_aesni_final(&state, tag_xor, req->assoclen, cryptlen); crypto_aegis256_aesni_final(&state, tag_xor, req->assoclen, cryptlen);
kernel_fpu_end(); kernel_fpu_end();

47
arch/x86/crypto/aesni-intel_glue.c
View File

@ -175,26 +175,18 @@ asmlinkage void aesni_gcm_finalize(void *ctx,
struct gcm_context_data *gdata, struct gcm_context_data *gdata,
u8 *auth_tag, unsigned long auth_tag_len); u8 *auth_tag, unsigned long auth_tag_len);
static struct aesni_gcm_tfm_s { static const struct aesni_gcm_tfm_s {
void (*init)(void *ctx, void (*init)(void *ctx, struct gcm_context_data *gdata, u8 *iv,
struct gcm_context_data *gdata, u8 *hash_subkey, const u8 *aad, unsigned long aad_len);
u8 *iv, void (*enc_update)(void *ctx, struct gcm_context_data *gdata, u8 *out,
u8 *hash_subkey, const u8 *aad, const u8 *in, unsigned long plaintext_len);
unsigned long aad_len); void (*dec_update)(void *ctx, struct gcm_context_data *gdata, u8 *out,
void (*enc_update)(void *ctx, const u8 *in, unsigned long ciphertext_len);
struct gcm_context_data *gdata, u8 *out, void (*finalize)(void *ctx, struct gcm_context_data *gdata,
const u8 *in, u8 *auth_tag, unsigned long auth_tag_len);
unsigned long plaintext_len);
void (*dec_update)(void *ctx,
struct gcm_context_data *gdata, u8 *out,
const u8 *in,
unsigned long ciphertext_len);
void (*finalize)(void *ctx,
struct gcm_context_data *gdata,
u8 *auth_tag, unsigned long auth_tag_len);
} *aesni_gcm_tfm; } *aesni_gcm_tfm;
struct aesni_gcm_tfm_s aesni_gcm_tfm_sse = { static const struct aesni_gcm_tfm_s aesni_gcm_tfm_sse = {
.init = &aesni_gcm_init, .init = &aesni_gcm_init,
.enc_update = &aesni_gcm_enc_update, .enc_update = &aesni_gcm_enc_update,
.dec_update = &aesni_gcm_dec_update, .dec_update = &aesni_gcm_dec_update,
@ -243,7 +235,7 @@ asmlinkage void aesni_gcm_dec_avx_gen2(void *ctx,
const u8 *aad, unsigned long aad_len, const u8 *aad, unsigned long aad_len,
u8 *auth_tag, unsigned long auth_tag_len); u8 *auth_tag, unsigned long auth_tag_len);
struct aesni_gcm_tfm_s aesni_gcm_tfm_avx_gen2 = { static const struct aesni_gcm_tfm_s aesni_gcm_tfm_avx_gen2 = {
.init = &aesni_gcm_init_avx_gen2, .init = &aesni_gcm_init_avx_gen2,
.enc_update = &aesni_gcm_enc_update_avx_gen2, .enc_update = &aesni_gcm_enc_update_avx_gen2,
.dec_update = &aesni_gcm_dec_update_avx_gen2, .dec_update = &aesni_gcm_dec_update_avx_gen2,
@ -288,7 +280,7 @@ asmlinkage void aesni_gcm_dec_avx_gen4(void *ctx,
const u8 *aad, unsigned long aad_len, const u8 *aad, unsigned long aad_len,
u8 *auth_tag, unsigned long auth_tag_len); u8 *auth_tag, unsigned long auth_tag_len);
struct aesni_gcm_tfm_s aesni_gcm_tfm_avx_gen4 = { static const struct aesni_gcm_tfm_s aesni_gcm_tfm_avx_gen4 = {
.init = &aesni_gcm_init_avx_gen4, .init = &aesni_gcm_init_avx_gen4,
.enc_update = &aesni_gcm_enc_update_avx_gen4, .enc_update = &aesni_gcm_enc_update_avx_gen4,
.dec_update = &aesni_gcm_dec_update_avx_gen4, .dec_update = &aesni_gcm_dec_update_avx_gen4,
@ -778,7 +770,7 @@ static int gcmaes_crypt_by_sg(bool enc, struct aead_request *req,
{ {
struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct crypto_aead *tfm = crypto_aead_reqtfm(req);
unsigned long auth_tag_len = crypto_aead_authsize(tfm); unsigned long auth_tag_len = crypto_aead_authsize(tfm);
struct aesni_gcm_tfm_s *gcm_tfm = aesni_gcm_tfm; const struct aesni_gcm_tfm_s *gcm_tfm = aesni_gcm_tfm;
struct gcm_context_data data AESNI_ALIGN_ATTR; struct gcm_context_data data AESNI_ALIGN_ATTR;
struct scatter_walk dst_sg_walk = {}; struct scatter_walk dst_sg_walk = {};
unsigned long left = req->cryptlen; unsigned long left = req->cryptlen;
@ -821,11 +813,14 @@ static int gcmaes_crypt_by_sg(bool enc, struct aead_request *req,
scatterwalk_map_and_copy(assoc, req->src, 0, assoclen, 0); scatterwalk_map_and_copy(assoc, req->src, 0, assoclen, 0);
} }
src_sg = scatterwalk_ffwd(src_start, req->src, req->assoclen); if (left) {
scatterwalk_start(&src_sg_walk, src_sg); src_sg = scatterwalk_ffwd(src_start, req->src, req->assoclen);
if (req->src != req->dst) { scatterwalk_start(&src_sg_walk, src_sg);
dst_sg = scatterwalk_ffwd(dst_start, req->dst, req->assoclen); if (req->src != req->dst) {
scatterwalk_start(&dst_sg_walk, dst_sg); dst_sg = scatterwalk_ffwd(dst_start, req->dst,
req->assoclen);
scatterwalk_start(&dst_sg_walk, dst_sg);
}
} }
kernel_fpu_begin(); kernel_fpu_begin();

752
arch/x86/crypto/crct10dif-pcl-asm_64.S
View File

@ -43,609 +43,291 @@
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING # LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS # NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. # SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
########################################################################
# Function API:
# UINT16 crc_t10dif_pcl(
# UINT16 init_crc, //initial CRC value, 16 bits
# const unsigned char *buf, //buffer pointer to calculate CRC on
# UINT64 len //buffer length in bytes (64-bit data)
# );
# #
# Reference paper titled "Fast CRC Computation for Generic # Reference paper titled "Fast CRC Computation for Generic
# Polynomials Using PCLMULQDQ Instruction" # Polynomials Using PCLMULQDQ Instruction"
# URL: http://www.intel.com/content/dam/www/public/us/en/documents # URL: http://www.intel.com/content/dam/www/public/us/en/documents
# /white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf # /white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf
# #
#
#include <linux/linkage.h> #include <linux/linkage.h>
.text .text
#define arg1 %rdi #define init_crc %edi
#define arg2 %rsi #define buf %rsi
#define arg3 %rdx #define len %rdx
#define arg1_low32 %edi #define FOLD_CONSTS %xmm10
#define BSWAP_MASK %xmm11
ENTRY(crc_t10dif_pcl) # Fold reg1, reg2 into the next 32 data bytes, storing the result back into
# reg1, reg2.
.macro fold_32_bytes offset, reg1, reg2
movdqu \offset(buf), %xmm9
movdqu \offset+16(buf), %xmm12
pshufb BSWAP_MASK, %xmm9
pshufb BSWAP_MASK, %xmm12
movdqa \reg1, %xmm8
movdqa \reg2, %xmm13
pclmulqdq $0x00, FOLD_CONSTS, \reg1
pclmulqdq $0x11, FOLD_CONSTS, %xmm8
pclmulqdq $0x00, FOLD_CONSTS, \reg2
pclmulqdq $0x11, FOLD_CONSTS, %xmm13
pxor %xmm9 , \reg1
xorps %xmm8 , \reg1
pxor %xmm12, \reg2
xorps %xmm13, \reg2
.endm
# Fold src_reg into dst_reg.
.macro fold_16_bytes src_reg, dst_reg
movdqa \src_reg, %xmm8
pclmulqdq $0x11, FOLD_CONSTS, \src_reg
pclmulqdq $0x00, FOLD_CONSTS, %xmm8
pxor %xmm8, \dst_reg
xorps \src_reg, \dst_reg
.endm
#
# u16 crc_t10dif_pcl(u16 init_crc, const *u8 buf, size_t len);
#
# Assumes len >= 16.
#
.align 16 .align 16
ENTRY(crc_t10dif_pcl)
# adjust the 16-bit initial_crc value, scale it to 32 bits movdqa .Lbswap_mask(%rip), BSWAP_MASK
shl $16, arg1_low32
# Allocate Stack Space # For sizes less than 256 bytes, we can't fold 128 bytes at a time.
mov %rsp, %rcx cmp $256, len
sub $16*2, %rsp jl .Lless_than_256_bytes
# align stack to 16 byte boundary
and $~(0x10 - 1), %rsp
# check if smaller than 256 # Load the first 128 data bytes. Byte swapping is necessary to make the
cmp $256, arg3 # bit order match the polynomial coefficient order.
movdqu 16*0(buf), %xmm0
movdqu 16*1(buf), %xmm1
movdqu 16*2(buf), %xmm2
movdqu 16*3(buf), %xmm3
movdqu 16*4(buf), %xmm4
movdqu 16*5(buf), %xmm5
movdqu 16*6(buf), %xmm6
movdqu 16*7(buf), %xmm7
add $128, buf
pshufb BSWAP_MASK, %xmm0
pshufb BSWAP_MASK, %xmm1
pshufb BSWAP_MASK, %xmm2
pshufb BSWAP_MASK, %xmm3
pshufb BSWAP_MASK, %xmm4
pshufb BSWAP_MASK, %xmm5
pshufb BSWAP_MASK, %xmm6
pshufb BSWAP_MASK, %xmm7
# for sizes less than 128, we can't fold 64B at a time... # XOR the first 16 data *bits* with the initial CRC value.
jl _less_than_128 pxor %xmm8, %xmm8
pinsrw $7, init_crc, %xmm8
pxor %xmm8, %xmm0
movdqa .Lfold_across_128_bytes_consts(%rip), FOLD_CONSTS
# load the initial crc value # Subtract 128 for the 128 data bytes just consumed. Subtract another
movd arg1_low32, %xmm10 # initial crc # 128 to simplify the termination condition of the following loop.
sub $256, len
# crc value does not need to be byte-reflected, but it needs # While >= 128 data bytes remain (not counting xmm0-7), fold the 128
# to be moved to the high part of the register. # bytes xmm0-7 into them, storing the result back into xmm0-7.
# because data will be byte-reflected and will align with .Lfold_128_bytes_loop:
# initial crc at correct place. fold_32_bytes 0, %xmm0, %xmm1
pslldq $12, %xmm10 fold_32_bytes 32, %xmm2, %xmm3
fold_32_bytes 64, %xmm4, %xmm5
fold_32_bytes 96, %xmm6, %xmm7
add $128, buf
sub $128, len
jge .Lfold_128_bytes_loop
movdqa SHUF_MASK(%rip), %xmm11 # Now fold the 112 bytes in xmm0-xmm6 into the 16 bytes in xmm7.
# receive the initial 64B data, xor the initial crc value
movdqu 16*0(arg2), %xmm0
movdqu 16*1(arg2), %xmm1
movdqu 16*2(arg2), %xmm2
movdqu 16*3(arg2), %xmm3
movdqu 16*4(arg2), %xmm4
movdqu 16*5(arg2), %xmm5
movdqu 16*6(arg2), %xmm6
movdqu 16*7(arg2), %xmm7
pshufb %xmm11, %xmm0 # Fold across 64 bytes.
# XOR the initial_crc value movdqa .Lfold_across_64_bytes_consts(%rip), FOLD_CONSTS
pxor %xmm10, %xmm0 fold_16_bytes %xmm0, %xmm4
pshufb %xmm11, %xmm1 fold_16_bytes %xmm1, %xmm5
pshufb %xmm11, %xmm2 fold_16_bytes %xmm2, %xmm6
pshufb %xmm11, %xmm3 fold_16_bytes %xmm3, %xmm7
pshufb %xmm11, %xmm4 # Fold across 32 bytes.
pshufb %xmm11, %xmm5 movdqa .Lfold_across_32_bytes_consts(%rip), FOLD_CONSTS
pshufb %xmm11, %xmm6 fold_16_bytes %xmm4, %xmm6
pshufb %xmm11, %xmm7 fold_16_bytes %xmm5, %xmm7
# Fold across 16 bytes.
movdqa .Lfold_across_16_bytes_consts(%rip), FOLD_CONSTS
fold_16_bytes %xmm6, %xmm7
movdqa rk3(%rip), %xmm10 #xmm10 has rk3 and rk4 # Add 128 to get the correct number of data bytes remaining in 0...127
#imm value of pclmulqdq instruction # (not counting xmm7), following the previous extra subtraction by 128.
#will determine which constant to use # Then subtract 16 to simplify the termination condition of the
# following loop.
add $128-16, len
################################################################# # While >= 16 data bytes remain (not counting xmm7), fold the 16 bytes
# we subtract 256 instead of 128 to save one instruction from the loop # xmm7 into them, storing the result back into xmm7.
sub $256, arg3 jl .Lfold_16_bytes_loop_done
.Lfold_16_bytes_loop:
# at this section of the code, there is 64*x+y (0<=y<64) bytes of
# buffer. The _fold_64_B_loop will fold 64B at a time
# until we have 64+y Bytes of buffer
# fold 64B at a time. This section of the code folds 4 xmm
# registers in parallel
_fold_64_B_loop:
# update the buffer pointer
add $128, arg2 # buf += 64#
movdqu 16*0(arg2), %xmm9
movdqu 16*1(arg2), %xmm12
pshufb %xmm11, %xmm9
pshufb %xmm11, %xmm12
movdqa %xmm0, %xmm8
movdqa %xmm1, %xmm13
pclmulqdq $0x0 , %xmm10, %xmm0
pclmulqdq $0x11, %xmm10, %xmm8
pclmulqdq $0x0 , %xmm10, %xmm1
pclmulqdq $0x11, %xmm10, %xmm13
pxor %xmm9 , %xmm0
xorps %xmm8 , %xmm0
pxor %xmm12, %xmm1
xorps %xmm13, %xmm1
movdqu 16*2(arg2), %xmm9
movdqu 16*3(arg2), %xmm12
pshufb %xmm11, %xmm9
pshufb %xmm11, %xmm12
movdqa %xmm2, %xmm8
movdqa %xmm3, %xmm13
pclmulqdq $0x0, %xmm10, %xmm2
pclmulqdq $0x11, %xmm10, %xmm8
pclmulqdq $0x0, %xmm10, %xmm3
pclmulqdq $0x11, %xmm10, %xmm13
pxor %xmm9 , %xmm2
xorps %xmm8 , %xmm2
pxor %xmm12, %xmm3
xorps %xmm13, %xmm3
movdqu 16*4(arg2), %xmm9
movdqu 16*5(arg2), %xmm12
pshufb %xmm11, %xmm9
pshufb %xmm11, %xmm12
movdqa %xmm4, %xmm8
movdqa %xmm5, %xmm13
pclmulqdq $0x0, %xmm10, %xmm4
pclmulqdq $0x11, %xmm10, %xmm8
pclmulqdq $0x0, %xmm10, %xmm5
pclmulqdq $0x11, %xmm10, %xmm13
pxor %xmm9 , %xmm4
xorps %xmm8 , %xmm4
pxor %xmm12, %xmm5
xorps %xmm13, %xmm5
movdqu 16*6(arg2), %xmm9
movdqu 16*7(arg2), %xmm12
pshufb %xmm11, %xmm9
pshufb %xmm11, %xmm12
movdqa %xmm6 , %xmm8
movdqa %xmm7 , %xmm13
pclmulqdq $0x0 , %xmm10, %xmm6
pclmulqdq $0x11, %xmm10, %xmm8
pclmulqdq $0x0 , %xmm10, %xmm7
pclmulqdq $0x11, %xmm10, %xmm13
pxor %xmm9 , %xmm6
xorps %xmm8 , %xmm6
pxor %xmm12, %xmm7
xorps %xmm13, %xmm7
sub $128, arg3
# check if there is another 64B in the buffer to be able to fold
jge _fold_64_B_loop
##################################################################
add $128, arg2
# at this point, the buffer pointer is pointing at the last y Bytes
# of the buffer the 64B of folded data is in 4 of the xmm
# registers: xmm0, xmm1, xmm2, xmm3
# fold the 8 xmm registers to 1 xmm register with different constants
movdqa rk9(%rip), %xmm10
movdqa %xmm0, %xmm8
pclmulqdq $0x11, %xmm10, %xmm0
pclmulqdq $0x0 , %xmm10, %xmm8
pxor %xmm8, %xmm7
xorps %xmm0, %xmm7
movdqa rk11(%rip), %xmm10
movdqa %xmm1, %xmm8
pclmulqdq $0x11, %xmm10, %xmm1
pclmulqdq $0x0 , %xmm10, %xmm8
pxor %xmm8, %xmm7
xorps %xmm1, %xmm7
movdqa rk13(%rip), %xmm10
movdqa %xmm2, %xmm8
pclmulqdq $0x11, %xmm10, %xmm2
pclmulqdq $0x0 , %xmm10, %xmm8
pxor %xmm8, %xmm7
pxor %xmm2, %xmm7
movdqa rk15(%rip), %xmm10
movdqa %xmm3, %xmm8
pclmulqdq $0x11, %xmm10, %xmm3
pclmulqdq $0x0 , %xmm10, %xmm8
pxor %xmm8, %xmm7
xorps %xmm3, %xmm7
movdqa rk17(%rip), %xmm10
movdqa %xmm4, %xmm8
pclmulqdq $0x11, %xmm10, %xmm4
pclmulqdq $0x0 , %xmm10, %xmm8
pxor %xmm8, %xmm7
pxor %xmm4, %xmm7
movdqa rk19(%rip), %xmm10
movdqa %xmm5, %xmm8
pclmulqdq $0x11, %xmm10, %xmm5
pclmulqdq $0x0 , %xmm10, %xmm8
pxor %xmm8, %xmm7
xorps %xmm5, %xmm7
movdqa rk1(%rip), %xmm10 #xmm10 has rk1 and rk2
#imm value of pclmulqdq instruction
#will determine which constant to use
movdqa %xmm6, %xmm8
pclmulqdq $0x11, %xmm10, %xmm6
pclmulqdq $0x0 , %xmm10, %xmm8
pxor %xmm8, %xmm7
pxor %xmm6, %xmm7
# instead of 64, we add 48 to the loop counter to save 1 instruction
# from the loop instead of a cmp instruction, we use the negative
# flag with the jl instruction
add $128-16, arg3
jl _final_reduction_for_128
# now we have 16+y bytes left to reduce. 16 Bytes is in register xmm7
# and the rest is in memory. We can fold 16 bytes at a time if y>=16
# continue folding 16B at a time
_16B_reduction_loop:
movdqa %xmm7, %xmm8 movdqa %xmm7, %xmm8
pclmulqdq $0x11, %xmm10, %xmm7 pclmulqdq $0x11, FOLD_CONSTS, %xmm7
pclmulqdq $0x0 , %xmm10, %xmm8 pclmulqdq $0x00, FOLD_CONSTS, %xmm8
pxor %xmm8, %xmm7 pxor %xmm8, %xmm7
movdqu (arg2), %xmm0 movdqu (buf), %xmm0
pshufb %xmm11, %xmm0 pshufb BSWAP_MASK, %xmm0
pxor %xmm0 , %xmm7 pxor %xmm0 , %xmm7
add $16, arg2 add $16, buf
sub $16, arg3 sub $16, len
# instead of a cmp instruction, we utilize the flags with the jge .Lfold_16_bytes_loop
# jge instruction equivalent of: cmp arg3, 16-16
# check if there is any more 16B in the buffer to be able to fold
jge _16B_reduction_loop
#now we have 16+z bytes left to reduce, where 0<= z < 16. .Lfold_16_bytes_loop_done:
#first, we reduce the data in the xmm7 register # Add 16 to get the correct number of data bytes remaining in 0...15
# (not counting xmm7), following the previous extra subtraction by 16.
add $16, len
je .Lreduce_final_16_bytes
.Lhandle_partial_segment:
# Reduce the last '16 + len' bytes where 1 <= len <= 15 and the first 16
# bytes are in xmm7 and the rest are the remaining data in 'buf'. To do
# this without needing a fold constant for each possible 'len', redivide
# the bytes into a first chunk of 'len' bytes and a second chunk of 16
# bytes, then fold the first chunk into the second.
_final_reduction_for_128:
# check if any more data to fold. If not, compute the CRC of
# the final 128 bits
add $16, arg3
je _128_done
# here we are getting data that is less than 16 bytes.
# since we know that there was data before the pointer, we can
# offset the input pointer before the actual point, to receive
# exactly 16 bytes. after that the registers need to be adjusted.
_get_last_two_xmms:
movdqa %xmm7, %xmm2 movdqa %xmm7, %xmm2
movdqu -16(arg2, arg3), %xmm1 # xmm1 = last 16 original data bytes
pshufb %xmm11, %xmm1 movdqu -16(buf, len), %xmm1
pshufb BSWAP_MASK, %xmm1
# get rid of the extra data that was loaded before # xmm2 = high order part of second chunk: xmm7 left-shifted by 'len' bytes.
# load the shift constant lea .Lbyteshift_table+16(%rip), %rax
lea pshufb_shf_table+16(%rip), %rax sub len, %rax
sub arg3, %rax
movdqu (%rax), %xmm0 movdqu (%rax), %xmm0
# shift xmm2 to the left by arg3 bytes
pshufb %xmm0, %xmm2 pshufb %xmm0, %xmm2
# shift xmm7 to the right by 16-arg3 bytes # xmm7 = first chunk: xmm7 right-shifted by '16-len' bytes.
pxor mask1(%rip), %xmm0 pxor .Lmask1(%rip), %xmm0
pshufb %xmm0, %xmm7 pshufb %xmm0, %xmm7
# xmm1 = second chunk: 'len' bytes from xmm1 (low-order bytes),
# then '16-len' bytes from xmm2 (high-order bytes).
pblendvb %xmm2, %xmm1 #xmm0 is implicit pblendvb %xmm2, %xmm1 #xmm0 is implicit
# fold 16 Bytes # Fold the first chunk into the second chunk, storing the result in xmm7.
movdqa %xmm1, %xmm2
movdqa %xmm7, %xmm8 movdqa %xmm7, %xmm8
pclmulqdq $0x11, %xmm10, %xmm7 pclmulqdq $0x11, FOLD_CONSTS, %xmm7
pclmulqdq $0x0 , %xmm10, %xmm8 pclmulqdq $0x00, FOLD_CONSTS, %xmm8
pxor %xmm8, %xmm7 pxor %xmm8, %xmm7
pxor %xmm2, %xmm7 pxor %xmm1, %xmm7
_128_done: .Lreduce_final_16_bytes:
# compute crc of a 128-bit value # Reduce the 128-bit value M(x), stored in xmm7, to the final 16-bit CRC
movdqa rk5(%rip), %xmm10 # rk5 and rk6 in xmm10
# Load 'x^48 * (x^48 mod G(x))' and 'x^48 * (x^80 mod G(x))'.
movdqa .Lfinal_fold_consts(%rip), FOLD_CONSTS
# Fold the high 64 bits into the low 64 bits, while also multiplying by
# x^64. This produces a 128-bit value congruent to x^64 * M(x) and
# whose low 48 bits are 0.
movdqa %xmm7, %xmm0 movdqa %xmm7, %xmm0
pclmulqdq $0x11, FOLD_CONSTS, %xmm7 # high bits * x^48 * (x^80 mod G(x))
pslldq $8, %xmm0
pxor %xmm0, %xmm7 # + low bits * x^64
#64b fold # Fold the high 32 bits into the low 96 bits. This produces a 96-bit
pclmulqdq $0x1, %xmm10, %xmm7 # value congruent to x^64 * M(x) and whose low 48 bits are 0.
pslldq $8 , %xmm0
pxor %xmm0, %xmm7
#32b fold
movdqa %xmm7, %xmm0 movdqa %xmm7, %xmm0
pand .Lmask2(%rip), %xmm0 # zero high 32 bits
psrldq $12, %xmm7 # extract high 32 bits
pclmulqdq $0x00, FOLD_CONSTS, %xmm7 # high 32 bits * x^48 * (x^48 mod G(x))
pxor %xmm0, %xmm7 # + low bits
pand mask2(%rip), %xmm0 # Load G(x) and floor(x^48 / G(x)).
movdqa .Lbarrett_reduction_consts(%rip), FOLD_CONSTS
psrldq $12, %xmm7 # Use Barrett reduction to compute the final CRC value.
pclmulqdq $0x10, %xmm10, %xmm7
pxor %xmm0, %xmm7
#barrett reduction
_barrett:
movdqa rk7(%rip), %xmm10 # rk7 and rk8 in xmm10
movdqa %xmm7, %xmm0 movdqa %xmm7, %xmm0
pclmulqdq $0x01, %xmm10, %xmm7 pclmulqdq $0x11, FOLD_CONSTS, %xmm7 # high 32 bits * floor(x^48 / G(x))
pslldq $4, %xmm7 psrlq $32, %xmm7 # /= x^32
pclmulqdq $0x11, %xmm10, %xmm7 pclmulqdq $0x00, FOLD_CONSTS, %xmm7 # *= G(x)
psrlq $48, %xmm0
pxor %xmm7, %xmm0 # + low 16 nonzero bits
# Final CRC value (x^16 * M(x)) mod G(x) is in low 16 bits of xmm0.
pslldq $4, %xmm7 pextrw $0, %xmm0, %eax
pxor %xmm0, %xmm7
pextrd $1, %xmm7, %eax
_cleanup:
# scale the result back to 16 bits
shr $16, %eax
mov %rcx, %rsp
ret ret
########################################################################
.align 16 .align 16
_less_than_128: .Lless_than_256_bytes:
# Checksumming a buffer of length 16...255 bytes
# check if there is enough buffer to be able to fold 16B at a time # Load the first 16 data bytes.
cmp $32, arg3 movdqu (buf), %xmm7
jl _less_than_32 pshufb BSWAP_MASK, %xmm7
movdqa SHUF_MASK(%rip), %xmm11 add $16, buf
# now if there is, load the constants # XOR the first 16 data *bits* with the initial CRC value.
movdqa rk1(%rip), %xmm10 # rk1 and rk2 in xmm10 pxor %xmm0, %xmm0
pinsrw $7, init_crc, %xmm0
movd arg1_low32, %xmm0 # get the initial crc value
pslldq $12, %xmm0 # align it to its correct place
movdqu (arg2), %xmm7 # load the plaintext
pshufb %xmm11, %xmm7 # byte-reflect the plaintext
pxor %xmm0, %xmm7 pxor %xmm0, %xmm7
movdqa .Lfold_across_16_bytes_consts(%rip), FOLD_CONSTS
# update the buffer pointer cmp $16, len
add $16, arg2 je .Lreduce_final_16_bytes # len == 16
sub $32, len
# update the counter. subtract 32 instead of 16 to save one jge .Lfold_16_bytes_loop # 32 <= len <= 255
# instruction from the loop add $16, len
sub $32, arg3 jmp .Lhandle_partial_segment # 17 <= len <= 31
jmp _16B_reduction_loop
.align 16
_less_than_32:
# mov initial crc to the return value. this is necessary for
# zero-length buffers.
mov arg1_low32, %eax
test arg3, arg3
je _cleanup
movdqa SHUF_MASK(%rip), %xmm11
movd arg1_low32, %xmm0 # get the initial crc value
pslldq $12, %xmm0 # align it to its correct place
cmp $16, arg3
je _exact_16_left
jl _less_than_16_left
movdqu (arg2), %xmm7 # load the plaintext
pshufb %xmm11, %xmm7 # byte-reflect the plaintext
pxor %xmm0 , %xmm7 # xor the initial crc value
add $16, arg2
sub $16, arg3
movdqa rk1(%rip), %xmm10 # rk1 and rk2 in xmm10
jmp _get_last_two_xmms
.align 16
_less_than_16_left:
# use stack space to load data less than 16 bytes, zero-out
# the 16B in memory first.
pxor %xmm1, %xmm1
mov %rsp, %r11
movdqa %xmm1, (%r11)
cmp $4, arg3
jl _only_less_than_4
# backup the counter value
mov arg3, %r9
cmp $8, arg3
jl _less_than_8_left
# load 8 Bytes
mov (arg2), %rax
mov %rax, (%r11)
add $8, %r11
sub $8, arg3
add $8, arg2
_less_than_8_left:
cmp $4, arg3
jl _less_than_4_left
# load 4 Bytes
mov (arg2), %eax
mov %eax, (%r11)
add $4, %r11
sub $4, arg3
add $4, arg2
_less_than_4_left:
cmp $2, arg3
jl _less_than_2_left
# load 2 Bytes
mov (arg2), %ax
mov %ax, (%r11)
add $2, %r11
sub $2, arg3
add $2, arg2
_less_than_2_left:
cmp $1, arg3
jl _zero_left
# load 1 Byte
mov (arg2), %al
mov %al, (%r11)
_zero_left:
movdqa (%rsp), %xmm7
pshufb %xmm11, %xmm7
pxor %xmm0 , %xmm7 # xor the initial crc value
# shl r9, 4
lea pshufb_shf_table+16(%rip), %rax
sub %r9, %rax
movdqu (%rax), %xmm0
pxor mask1(%rip), %xmm0
pshufb %xmm0, %xmm7
jmp _128_done
.align 16
_exact_16_left:
movdqu (arg2), %xmm7
pshufb %xmm11, %xmm7
pxor %xmm0 , %xmm7 # xor the initial crc value
jmp _128_done
_only_less_than_4:
cmp $3, arg3
jl _only_less_than_3
# load 3 Bytes
mov (arg2), %al
mov %al, (%r11)
mov 1(arg2), %al
mov %al, 1(%r11)
mov 2(arg2), %al
mov %al, 2(%r11)
movdqa (%rsp), %xmm7
pshufb %xmm11, %xmm7
pxor %xmm0 , %xmm7 # xor the initial crc value
psrldq $5, %xmm7
jmp _barrett
_only_less_than_3:
cmp $2, arg3
jl _only_less_than_2
# load 2 Bytes
mov (arg2), %al
mov %al, (%r11)
mov 1(arg2), %al
mov %al, 1(%r11)
movdqa (%rsp), %xmm7
pshufb %xmm11, %xmm7
pxor %xmm0 , %xmm7 # xor the initial crc value
psrldq $6, %xmm7
jmp _barrett
_only_less_than_2:
# load 1 Byte
mov (arg2), %al
mov %al, (%r11)
movdqa (%rsp), %xmm7
pshufb %xmm11, %xmm7
pxor %xmm0 , %xmm7 # xor the initial crc value
psrldq $7, %xmm7
jmp _barrett
ENDPROC(crc_t10dif_pcl) ENDPROC(crc_t10dif_pcl)
.section .rodata, "a", @progbits .section .rodata, "a", @progbits
.align 16 .align 16
# precomputed constants
# these constants are precomputed from the poly:
# 0x8bb70000 (0x8bb7 scaled to 32 bits)
# Q = 0x18BB70000
# rk1 = 2^(32*3) mod Q << 32
# rk2 = 2^(32*5) mod Q << 32
# rk3 = 2^(32*15) mod Q << 32
# rk4 = 2^(32*17) mod Q << 32
# rk5 = 2^(32*3) mod Q << 32
# rk6 = 2^(32*2) mod Q << 32
# rk7 = floor(2^64/Q)
# rk8 = Q
rk1:
.quad 0x2d56000000000000
rk2:
.quad 0x06df000000000000
rk3:
.quad 0x9d9d000000000000
rk4:
.quad 0x7cf5000000000000
rk5:
.quad 0x2d56000000000000
rk6:
.quad 0x1368000000000000
rk7:
.quad 0x00000001f65a57f8
rk8:
.quad 0x000000018bb70000
rk9:
.quad 0xceae000000000000
rk10:
.quad 0xbfd6000000000000
rk11:
.quad 0x1e16000000000000
rk12:
.quad 0x713c000000000000
rk13:
.quad 0xf7f9000000000000
rk14:
.quad 0x80a6000000000000
rk15:
.quad 0x044c000000000000
rk16:
.quad 0xe658000000000000
rk17:
.quad 0xad18000000000000
rk18:
.quad 0xa497000000000000
rk19:
.quad 0x6ee3000000000000
rk20:
.quad 0xe7b5000000000000
# Fold constants precomputed from the polynomial 0x18bb7
# G(x) = x^16 + x^15 + x^11 + x^9 + x^8 + x^7 + x^5 + x^4 + x^2 + x^1 + x^0
.Lfold_across_128_bytes_consts:
.quad 0x0000000000006123 # x^(8*128) mod G(x)
.quad 0x0000000000002295 # x^(8*128+64) mod G(x)
.Lfold_across_64_bytes_consts:
.quad 0x0000000000001069 # x^(4*128) mod G(x)
.quad 0x000000000000dd31 # x^(4*128+64) mod G(x)
.Lfold_across_32_bytes_consts:
.quad 0x000000000000857d # x^(2*128) mod G(x)
.quad 0x0000000000007acc # x^(2*128+64) mod G(x)
.Lfold_across_16_bytes_consts:
.quad 0x000000000000a010 # x^(1*128) mod G(x)
.quad 0x0000000000001faa # x^(1*128+64) mod G(x)
.Lfinal_fold_consts:
.quad 0x1368000000000000 # x^48 * (x^48 mod G(x))
.quad 0x2d56000000000000 # x^48 * (x^80 mod G(x))
.Lbarrett_reduction_consts:
.quad 0x0000000000018bb7 # G(x)
.quad 0x00000001f65a57f8 # floor(x^48 / G(x))
.section .rodata.cst16.mask1, "aM", @progbits, 16 .section .rodata.cst16.mask1, "aM", @progbits, 16
.align 16 .align 16
mask1: .Lmask1:
.octa 0x80808080808080808080808080808080 .octa 0x80808080808080808080808080808080
.section .rodata.cst16.mask2, "aM", @progbits, 16 .section .rodata.cst16.mask2, "aM", @progbits, 16
.align 16 .align 16
mask2: .Lmask2:
.octa 0x00000000FFFFFFFFFFFFFFFFFFFFFFFF .octa 0x00000000FFFFFFFFFFFFFFFFFFFFFFFF
.section .rodata.cst16.SHUF_MASK, "aM", @progbits, 16 .section .rodata.cst16.bswap_mask, "aM", @progbits, 16
.align 16 .align 16
SHUF_MASK: .Lbswap_mask:
.octa 0x000102030405060708090A0B0C0D0E0F .octa 0x000102030405060708090A0B0C0D0E0F
.section .rodata.cst32.pshufb_shf_table, "aM", @progbits, 32 .section .rodata.cst32.byteshift_table, "aM", @progbits, 32
.align 32 .align 16
pshufb_shf_table: # For 1 <= len <= 15, the 16-byte vector beginning at &byteshift_table[16 - len]
# use these values for shift constants for the pshufb instruction # is the index vector to shift left by 'len' bytes, and is also {0x80, ...,
# different alignments result in values as shown: # 0x80} XOR the index vector to shift right by '16 - len' bytes.
# DDQ 0x008f8e8d8c8b8a898887868584838281 # shl 15 (16-1) / shr1 .Lbyteshift_table:
# DDQ 0x01008f8e8d8c8b8a8988878685848382 # shl 14 (16-3) / shr2 .byte 0x0, 0x81, 0x82, 0x83, 0x84, 0x85, 0x86, 0x87
# DDQ 0x0201008f8e8d8c8b8a89888786858483 # shl 13 (16-4) / shr3 .byte 0x88, 0x89, 0x8a, 0x8b, 0x8c, 0x8d, 0x8e, 0x8f
# DDQ 0x030201008f8e8d8c8b8a898887868584 # shl 12 (16-4) / shr4 .byte 0x0, 0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7
# DDQ 0x04030201008f8e8d8c8b8a8988878685 # shl 11 (16-5) / shr5 .byte 0x8, 0x9, 0xa, 0xb, 0xc, 0xd, 0xe , 0x0
# DDQ 0x0504030201008f8e8d8c8b8a89888786 # shl 10 (16-6) / shr6
# DDQ 0x060504030201008f8e8d8c8b8a898887 # shl 9 (16-7) / shr7
# DDQ 0x07060504030201008f8e8d8c8b8a8988 # shl 8 (16-8) / shr8
# DDQ 0x0807060504030201008f8e8d8c8b8a89 # shl 7 (16-9) / shr9
# DDQ 0x090807060504030201008f8e8d8c8b8a # shl 6 (16-10) / shr10
# DDQ 0x0a090807060504030201008f8e8d8c8b # shl 5 (16-11) / shr11
# DDQ 0x0b0a090807060504030201008f8e8d8c # shl 4 (16-12) / shr12
# DDQ 0x0c0b0a090807060504030201008f8e8d # shl 3 (16-13) / shr13
# DDQ 0x0d0c0b0a090807060504030201008f8e # shl 2 (16-14) / shr14
# DDQ 0x0e0d0c0b0a090807060504030201008f # shl 1 (16-15) / shr15
.octa 0x8f8e8d8c8b8a89888786858483828100
.octa 0x000e0d0c0b0a09080706050403020100

12
arch/x86/crypto/crct10dif-pclmul_glue.c
View File

@ -33,18 +33,12 @@
#include <asm/cpufeatures.h> #include <asm/cpufeatures.h>
#include <asm/cpu_device_id.h> #include <asm/cpu_device_id.h>
asmlinkage __u16 crc_t10dif_pcl(__u16 crc, const unsigned char *buf, asmlinkage u16 crc_t10dif_pcl(u16 init_crc, const u8 *buf, size_t len);
size_t len);
struct chksum_desc_ctx { struct chksum_desc_ctx {
__u16 crc; __u16 crc;
}; };
/*
* Steps through buffer one byte at at time, calculates reflected
* crc using table.
*/
static int chksum_init(struct shash_desc *desc) static int chksum_init(struct shash_desc *desc)
{ {
struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); struct chksum_desc_ctx *ctx = shash_desc_ctx(desc);
@ -59,7 +53,7 @@ static int chksum_update(struct shash_desc *desc, const u8 *data,
{ {
struct chksum_desc_ctx *ctx = shash_desc_ctx(desc); struct chksum_desc_ctx *ctx = shash_desc_ctx(desc);
if (irq_fpu_usable()) { if (length >= 16 && irq_fpu_usable()) {
kernel_fpu_begin(); kernel_fpu_begin();
ctx->crc = crc_t10dif_pcl(ctx->crc, data, length); ctx->crc = crc_t10dif_pcl(ctx->crc, data, length);
kernel_fpu_end(); kernel_fpu_end();
@ -79,7 +73,7 @@ static int chksum_final(struct shash_desc *desc, u8 *out)
static int __chksum_finup(__u16 *crcp, const u8 *data, unsigned int len, static int __chksum_finup(__u16 *crcp, const u8 *data, unsigned int len,
u8 *out) u8 *out)
{ {
if (irq_fpu_usable()) { if (len >= 16 && irq_fpu_usable()) {
kernel_fpu_begin(); kernel_fpu_begin();
*(__u16 *)out = crc_t10dif_pcl(*crcp, data, len); *(__u16 *)out = crc_t10dif_pcl(*crcp, data, len);
kernel_fpu_end(); kernel_fpu_end();

40
arch/x86/crypto/morus1280_glue.c
View File

@ -85,31 +85,20 @@ static void crypto_morus1280_glue_process_ad(
static void crypto_morus1280_glue_process_crypt(struct morus1280_state *state, static void crypto_morus1280_glue_process_crypt(struct morus1280_state *state,
struct morus1280_ops ops, struct morus1280_ops ops,
struct aead_request *req) struct skcipher_walk *walk)
{ {
struct skcipher_walk walk; while (walk->nbytes >= MORUS1280_BLOCK_SIZE) {
u8 *cursor_src, *cursor_dst; ops.crypt_blocks(state, walk->src.virt.addr,
unsigned int chunksize, base; walk->dst.virt.addr,
round_down(walk->nbytes,
MORUS1280_BLOCK_SIZE));
skcipher_walk_done(walk, walk->nbytes % MORUS1280_BLOCK_SIZE);
}
ops.skcipher_walk_init(&walk, req, false); if (walk->nbytes) {
ops.crypt_tail(state, walk->src.virt.addr, walk->dst.virt.addr,
while (walk.nbytes) { walk->nbytes);
cursor_src = walk.src.virt.addr; skcipher_walk_done(walk, 0);
cursor_dst = walk.dst.virt.addr;
chunksize = walk.nbytes;
ops.crypt_blocks(state, cursor_src, cursor_dst, chunksize);
base = chunksize & ~(MORUS1280_BLOCK_SIZE - 1);
cursor_src += base;
cursor_dst += base;
chunksize &= MORUS1280_BLOCK_SIZE - 1;
if (chunksize > 0)
ops.crypt_tail(state, cursor_src, cursor_dst,
chunksize);
skcipher_walk_done(&walk, 0);
} }
} }
@ -147,12 +136,15 @@ static void crypto_morus1280_glue_crypt(struct aead_request *req,
struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct morus1280_ctx *ctx = crypto_aead_ctx(tfm); struct morus1280_ctx *ctx = crypto_aead_ctx(tfm);
struct morus1280_state state; struct morus1280_state state;
struct skcipher_walk walk;
ops.skcipher_walk_init(&walk, req, true);
kernel_fpu_begin(); kernel_fpu_begin();
ctx->ops->init(&state, &ctx->key, req->iv); ctx->ops->init(&state, &ctx->key, req->iv);
crypto_morus1280_glue_process_ad(&state, ctx->ops, req->src, req->assoclen); crypto_morus1280_glue_process_ad(&state, ctx->ops, req->src, req->assoclen);
crypto_morus1280_glue_process_crypt(&state, ops, req); crypto_morus1280_glue_process_crypt(&state, ops, &walk);
ctx->ops->final(&state, tag_xor, req->assoclen, cryptlen); ctx->ops->final(&state, tag_xor, req->assoclen, cryptlen);
kernel_fpu_end(); kernel_fpu_end();

39
arch/x86/crypto/morus640_glue.c
View File

@ -85,31 +85,19 @@ static void crypto_morus640_glue_process_ad(
static void crypto_morus640_glue_process_crypt(struct morus640_state *state, static void crypto_morus640_glue_process_crypt(struct morus640_state *state,
struct morus640_ops ops, struct morus640_ops ops,
struct aead_request *req) struct skcipher_walk *walk)
{ {
struct skcipher_walk walk; while (walk->nbytes >= MORUS640_BLOCK_SIZE) {
u8 *cursor_src, *cursor_dst; ops.crypt_blocks(state, walk->src.virt.addr,
unsigned int chunksize, base; walk->dst.virt.addr,
round_down(walk->nbytes, MORUS640_BLOCK_SIZE));
skcipher_walk_done(walk, walk->nbytes % MORUS640_BLOCK_SIZE);
}
ops.skcipher_walk_init(&walk, req, false); if (walk->nbytes) {
ops.crypt_tail(state, walk->src.virt.addr, walk->dst.virt.addr,
while (walk.nbytes) { walk->nbytes);
cursor_src = walk.src.virt.addr; skcipher_walk_done(walk, 0);
cursor_dst = walk.dst.virt.addr;
chunksize = walk.nbytes;
ops.crypt_blocks(state, cursor_src, cursor_dst, chunksize);
base = chunksize & ~(MORUS640_BLOCK_SIZE - 1);
cursor_src += base;
cursor_dst += base;
chunksize &= MORUS640_BLOCK_SIZE - 1;
if (chunksize > 0)
ops.crypt_tail(state, cursor_src, cursor_dst,
chunksize);
skcipher_walk_done(&walk, 0);
} }
} }
@ -143,12 +131,15 @@ static void crypto_morus640_glue_crypt(struct aead_request *req,
struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct crypto_aead *tfm = crypto_aead_reqtfm(req);
struct morus640_ctx *ctx = crypto_aead_ctx(tfm); struct morus640_ctx *ctx = crypto_aead_ctx(tfm);
struct morus640_state state; struct morus640_state state;
struct skcipher_walk walk;
ops.skcipher_walk_init(&walk, req, true);
kernel_fpu_begin(); kernel_fpu_begin();
ctx->ops->init(&state, &ctx->key, req->iv); ctx->ops->init(&state, &ctx->key, req->iv);
crypto_morus640_glue_process_ad(&state, ctx->ops, req->src, req->assoclen); crypto_morus640_glue_process_ad(&state, ctx->ops, req->src, req->assoclen);
crypto_morus640_glue_process_crypt(&state, ops, req); crypto_morus640_glue_process_crypt(&state, ops, &walk);
ctx->ops->final(&state, tag_xor, req->assoclen, cryptlen); ctx->ops->final(&state, tag_xor, req->assoclen, cryptlen);
kernel_fpu_end(); kernel_fpu_end();

4
arch/x86/crypto/poly1305-sse2-x86_64.S
View File

@ -272,6 +272,10 @@ ENTRY(poly1305_block_sse2)
dec %rcx dec %rcx
jnz .Ldoblock jnz .Ldoblock
# Zeroing of key material
mov %rcx,0x00(%rsp)
mov %rcx,0x08(%rsp)
add $0x10,%rsp add $0x10,%rsp
pop %r12 pop %r12
pop %rbx pop %rbx

14
crypto/Kconfig
View File

@ -168,6 +168,16 @@ config CRYPTO_MANAGER_DISABLE_TESTS
Disable run-time self tests that normally take place at Disable run-time self tests that normally take place at
algorithm registration. algorithm registration.
config CRYPTO_MANAGER_EXTRA_TESTS
bool "Enable extra run-time crypto self tests"
depends on DEBUG_KERNEL && !CRYPTO_MANAGER_DISABLE_TESTS
help
Enable extra run-time self tests of registered crypto algorithms,
including randomized fuzz tests.
This is intended for developer use only, as these tests take much
longer to run than the normal self tests.
config CRYPTO_GF128MUL config CRYPTO_GF128MUL
tristate "GF(2^128) multiplication functions" tristate "GF(2^128) multiplication functions"
help help
@ -642,7 +652,7 @@ config CRYPTO_CRC32_PCLMUL
From Intel Westmere and AMD Bulldozer processor with SSE4.2 From Intel Westmere and AMD Bulldozer processor with SSE4.2
and PCLMULQDQ supported, the processor will support and PCLMULQDQ supported, the processor will support
CRC32 PCLMULQDQ implementation using hardware accelerated PCLMULQDQ CRC32 PCLMULQDQ implementation using hardware accelerated PCLMULQDQ
instruction. This option will create 'crc32-plcmul' module, instruction. This option will create 'crc32-pclmul' module,
which will enable any routine to use the CRC-32-IEEE 802.3 checksum which will enable any routine to use the CRC-32-IEEE 802.3 checksum
and gain better performance as compared with the table implementation. and gain better performance as compared with the table implementation.
@ -671,7 +681,7 @@ config CRYPTO_CRCT10DIF_PCLMUL
For x86_64 processors with SSE4.2 and PCLMULQDQ supported, For x86_64 processors with SSE4.2 and PCLMULQDQ supported,
CRC T10 DIF PCLMULQDQ computation can be hardware CRC T10 DIF PCLMULQDQ computation can be hardware
accelerated PCLMULQDQ instruction. This option will create accelerated PCLMULQDQ instruction. This option will create
'crct10dif-plcmul' module, which is faster when computing the 'crct10dif-pclmul' module, which is faster when computing the
crct10dif checksum as compared with the generic table implementation. crct10dif checksum as compared with the generic table implementation.
config CRYPTO_CRCT10DIF_VPMSUM config CRYPTO_CRCT10DIF_VPMSUM

4
crypto/aead.c
View File

@ -61,8 +61,10 @@ int crypto_aead_setkey(struct crypto_aead *tfm,
else else
err = crypto_aead_alg(tfm)->setkey(tfm, key, keylen); err = crypto_aead_alg(tfm)->setkey(tfm, key, keylen);
if (err) if (unlikely(err)) {
crypto_aead_set_flags(tfm, CRYPTO_TFM_NEED_KEY);
return err; return err;
}
crypto_aead_clear_flags(tfm, CRYPTO_TFM_NEED_KEY); crypto_aead_clear_flags(tfm, CRYPTO_TFM_NEED_KEY);
return 0; return 0;

7
crypto/aegis.h
View File

@ -1,14 +1,9 @@
/* SPDX-License-Identifier: GPL-2.0 */ /* SPDX-License-Identifier: GPL-2.0-or-later */
/* /*
* AEGIS common definitions * AEGIS common definitions
* *
* Copyright (c) 2018 Ondrej Mosnacek <omosnacek@gmail.com> * Copyright (c) 2018 Ondrej Mosnacek <omosnacek@gmail.com>
* Copyright (c) 2018 Red Hat, Inc. All rights reserved. * Copyright (c) 2018 Red Hat, Inc. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
* Software Foundation; either version 2 of the License, or (at your option)
* any later version.
*/ */
#ifndef _CRYPTO_AEGIS_H #ifndef _CRYPTO_AEGIS_H

20
crypto/aegis128.c
View File

@ -1,13 +1,9 @@
// SPDX-License-Identifier: GPL-2.0-or-later
/* /*
* The AEGIS-128 Authenticated-Encryption Algorithm * The AEGIS-128 Authenticated-Encryption Algorithm
* *
* Copyright (c) 2017-2018 Ondrej Mosnacek <omosnacek@gmail.com> * Copyright (c) 2017-2018 Ondrej Mosnacek <omosnacek@gmail.com>
* Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
* Software Foundation; either version 2 of the License, or (at your option)
* any later version.
*/ */
#include <crypto/algapi.h> #include <crypto/algapi.h>
@ -290,19 +286,19 @@ static void crypto_aegis128_process_crypt(struct aegis_state *state,
const struct aegis128_ops *ops) const struct aegis128_ops *ops)
{ {
struct skcipher_walk walk; struct skcipher_walk walk;
u8 *src, *dst;
unsigned int chunksize;
ops->skcipher_walk_init(&walk, req, false); ops->skcipher_walk_init(&walk, req, false);
while (walk.nbytes) { while (walk.nbytes) {
src = walk.src.virt.addr; unsigned int nbytes = walk.nbytes;
dst = walk.dst.virt.addr;
chunksize = walk.nbytes;
ops->crypt_chunk(state, dst, src, chunksize); if (nbytes < walk.total)
nbytes = round_down(nbytes, walk.stride);
skcipher_walk_done(&walk, 0); ops->crypt_chunk(state, walk.dst.virt.addr, walk.src.virt.addr,
nbytes);
skcipher_walk_done(&walk, walk.nbytes - nbytes);
} }
} }

20
crypto/aegis128l.c
View File

@ -1,13 +1,9 @@
// SPDX-License-Identifier: GPL-2.0-or-later
/* /*
* The AEGIS-128L Authenticated-Encryption Algorithm * The AEGIS-128L Authenticated-Encryption Algorithm
* *
* Copyright (c) 2017-2018 Ondrej Mosnacek <omosnacek@gmail.com> * Copyright (c) 2017-2018 Ondrej Mosnacek <omosnacek@gmail.com>
* Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
* Software Foundation; either version 2 of the License, or (at your option)
* any later version.
*/ */
#include <crypto/algapi.h> #include <crypto/algapi.h>
@ -353,19 +349,19 @@ static void crypto_aegis128l_process_crypt(struct aegis_state *state,
const struct aegis128l_ops *ops) const struct aegis128l_ops *ops)
{ {
struct skcipher_walk walk; struct skcipher_walk walk;
u8 *src, *dst;
unsigned int chunksize;
ops->skcipher_walk_init(&walk, req, false); ops->skcipher_walk_init(&walk, req, false);
while (walk.nbytes) { while (walk.nbytes) {
src = walk.src.virt.addr; unsigned int nbytes = walk.nbytes;
dst = walk.dst.virt.addr;
chunksize = walk.nbytes;
ops->crypt_chunk(state, dst, src, chunksize); if (nbytes < walk.total)
nbytes = round_down(nbytes, walk.stride);
skcipher_walk_done(&walk, 0); ops->crypt_chunk(state, walk.dst.virt.addr, walk.src.virt.addr,
nbytes);
skcipher_walk_done(&walk, walk.nbytes - nbytes);
} }
} }

20
crypto/aegis256.c
View File

@ -1,13 +1,9 @@
// SPDX-License-Identifier: GPL-2.0-or-later
/* /*
* The AEGIS-256 Authenticated-Encryption Algorithm * The AEGIS-256 Authenticated-Encryption Algorithm
* *
* Copyright (c) 2017-2018 Ondrej Mosnacek <omosnacek@gmail.com> * Copyright (c) 2017-2018 Ondrej Mosnacek <omosnacek@gmail.com>
* Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
* Software Foundation; either version 2 of the License, or (at your option)
* any later version.
*/ */
#include <crypto/algapi.h> #include <crypto/algapi.h>
@ -303,19 +299,19 @@ static void crypto_aegis256_process_crypt(struct aegis_state *state,
const struct aegis256_ops *ops) const struct aegis256_ops *ops)
{ {
struct skcipher_walk walk; struct skcipher_walk walk;
u8 *src, *dst;
unsigned int chunksize;
ops->skcipher_walk_init(&walk, req, false); ops->skcipher_walk_init(&walk, req, false);
while (walk.nbytes) { while (walk.nbytes) {
src = walk.src.virt.addr; unsigned int nbytes = walk.nbytes;
dst = walk.dst.virt.addr;
chunksize = walk.nbytes;
ops->crypt_chunk(state, dst, src, chunksize); if (nbytes < walk.total)
nbytes = round_down(nbytes, walk.stride);
skcipher_walk_done(&walk, 0); ops->crypt_chunk(state, walk.dst.virt.addr, walk.src.virt.addr,
nbytes);
skcipher_walk_done(&walk, walk.nbytes - nbytes);
} }
} }

36
crypto/af_alg.c
View File

@ -304,8 +304,6 @@ int af_alg_accept(struct sock *sk, struct socket *newsock, bool kern)
if (err) if (err)
goto unlock; goto unlock;
sk2->sk_family = PF_ALG;
if (nokey || !ask->refcnt++) if (nokey || !ask->refcnt++)
sock_hold(sk); sock_hold(sk);
ask->nokey_refcnt += nokey; ask->nokey_refcnt += nokey;
@ -382,7 +380,6 @@ static int alg_create(struct net *net, struct socket *sock, int protocol,
sock->ops = &alg_proto_ops; sock->ops = &alg_proto_ops;
sock_init_data(sock, sk); sock_init_data(sock, sk);
sk->sk_family = PF_ALG;
sk->sk_destruct = alg_sock_destruct; sk->sk_destruct = alg_sock_destruct;
return 0; return 0;
@ -427,12 +424,12 @@ int af_alg_make_sg(struct af_alg_sgl *sgl, struct iov_iter *iter, int len)
} }
EXPORT_SYMBOL_GPL(af_alg_make_sg); EXPORT_SYMBOL_GPL(af_alg_make_sg);
void af_alg_link_sg(struct af_alg_sgl *sgl_prev, struct af_alg_sgl *sgl_new) static void af_alg_link_sg(struct af_alg_sgl *sgl_prev,
struct af_alg_sgl *sgl_new)
{ {
sg_unmark_end(sgl_prev->sg + sgl_prev->npages - 1); sg_unmark_end(sgl_prev->sg + sgl_prev->npages - 1);
sg_chain(sgl_prev->sg, sgl_prev->npages + 1, sgl_new->sg); sg_chain(sgl_prev->sg, sgl_prev->npages + 1, sgl_new->sg);
} }
EXPORT_SYMBOL_GPL(af_alg_link_sg);
void af_alg_free_sg(struct af_alg_sgl *sgl) void af_alg_free_sg(struct af_alg_sgl *sgl)
{ {
@ -443,7 +440,7 @@ void af_alg_free_sg(struct af_alg_sgl *sgl)
} }
EXPORT_SYMBOL_GPL(af_alg_free_sg); EXPORT_SYMBOL_GPL(af_alg_free_sg);
int af_alg_cmsg_send(struct msghdr *msg, struct af_alg_control *con) static int af_alg_cmsg_send(struct msghdr *msg, struct af_alg_control *con)
{ {
struct cmsghdr *cmsg; struct cmsghdr *cmsg;
@ -482,7 +479,6 @@ int af_alg_cmsg_send(struct msghdr *msg, struct af_alg_control *con)
return 0; return 0;
} }
EXPORT_SYMBOL_GPL(af_alg_cmsg_send);
/** /**
* af_alg_alloc_tsgl - allocate the TX SGL * af_alg_alloc_tsgl - allocate the TX SGL
@ -490,7 +486,7 @@ EXPORT_SYMBOL_GPL(af_alg_cmsg_send);
* @sk socket of connection to user space * @sk socket of connection to user space
* @return: 0 upon success, < 0 upon error * @return: 0 upon success, < 0 upon error
*/ */
int af_alg_alloc_tsgl(struct sock *sk) static int af_alg_alloc_tsgl(struct sock *sk)
{ {
struct alg_sock *ask = alg_sk(sk); struct alg_sock *ask = alg_sk(sk);
struct af_alg_ctx *ctx = ask->private; struct af_alg_ctx *ctx = ask->private;
@ -519,7 +515,6 @@ int af_alg_alloc_tsgl(struct sock *sk)
return 0; return 0;
} }
EXPORT_SYMBOL_GPL(af_alg_alloc_tsgl);
/** /**
* aead_count_tsgl - Count number of TX SG entries * aead_count_tsgl - Count number of TX SG entries
@ -534,17 +529,17 @@ EXPORT_SYMBOL_GPL(af_alg_alloc_tsgl);
*/ */
unsigned int af_alg_count_tsgl(struct sock *sk, size_t bytes, size_t offset) unsigned int af_alg_count_tsgl(struct sock *sk, size_t bytes, size_t offset)
{ {
struct alg_sock *ask = alg_sk(sk); const struct alg_sock *ask = alg_sk(sk);
struct af_alg_ctx *ctx = ask->private; const struct af_alg_ctx *ctx = ask->private;
struct af_alg_tsgl *sgl, *tmp; const struct af_alg_tsgl *sgl;
unsigned int i; unsigned int i;
unsigned int sgl_count = 0; unsigned int sgl_count = 0;
if (!bytes) if (!bytes)
return 0; return 0;
list_for_each_entry_safe(sgl, tmp, &ctx->tsgl_list, list) { list_for_each_entry(sgl, &ctx->tsgl_list, list) {
struct scatterlist *sg = sgl->sg; const struct scatterlist *sg = sgl->sg;
for (i = 0; i < sgl->cur; i++) { for (i = 0; i < sgl->cur; i++) {
size_t bytes_count; size_t bytes_count;
@ -642,8 +637,7 @@ void af_alg_pull_tsgl(struct sock *sk, size_t used, struct scatterlist *dst,
} }
list_del(&sgl->list); list_del(&sgl->list);
sock_kfree_s(sk, sgl, sizeof(*sgl) + sizeof(sgl->sg[0]) * sock_kfree_s(sk, sgl, struct_size(sgl, sg, MAX_SGL_ENTS + 1));
(MAX_SGL_ENTS + 1));
} }
if (!ctx->used) if (!ctx->used)
@ -656,7 +650,7 @@ EXPORT_SYMBOL_GPL(af_alg_pull_tsgl);
* *
* @areq Request holding the TX and RX SGL * @areq Request holding the TX and RX SGL
*/ */
void af_alg_free_areq_sgls(struct af_alg_async_req *areq) static void af_alg_free_areq_sgls(struct af_alg_async_req *areq)
{ {
struct sock *sk = areq->sk; struct sock *sk = areq->sk;
struct alg_sock *ask = alg_sk(sk); struct alg_sock *ask = alg_sk(sk);
@ -685,7 +679,6 @@ void af_alg_free_areq_sgls(struct af_alg_async_req *areq)
sock_kfree_s(sk, tsgl, areq->tsgl_entries * sizeof(*tsgl)); sock_kfree_s(sk, tsgl, areq->tsgl_entries * sizeof(*tsgl));
} }
} }
EXPORT_SYMBOL_GPL(af_alg_free_areq_sgls);
/** /**
* af_alg_wait_for_wmem - wait for availability of writable memory * af_alg_wait_for_wmem - wait for availability of writable memory
@ -694,7 +687,7 @@ EXPORT_SYMBOL_GPL(af_alg_free_areq_sgls);
* @flags If MSG_DONTWAIT is set, then only report if function would sleep * @flags If MSG_DONTWAIT is set, then only report if function would sleep
* @return 0 when writable memory is available, < 0 upon error * @return 0 when writable memory is available, < 0 upon error
*/ */
int af_alg_wait_for_wmem(struct sock *sk, unsigned int flags) static int af_alg_wait_for_wmem(struct sock *sk, unsigned int flags)
{ {
DEFINE_WAIT_FUNC(wait, woken_wake_function); DEFINE_WAIT_FUNC(wait, woken_wake_function);
int err = -ERESTARTSYS; int err = -ERESTARTSYS;
@ -719,7 +712,6 @@ int af_alg_wait_for_wmem(struct sock *sk, unsigned int flags)
return err; return err;
} }
EXPORT_SYMBOL_GPL(af_alg_wait_for_wmem);
/** /**
* af_alg_wmem_wakeup - wakeup caller when writable memory is available * af_alg_wmem_wakeup - wakeup caller when writable memory is available
@ -788,8 +780,7 @@ EXPORT_SYMBOL_GPL(af_alg_wait_for_data);
* *
* @sk socket of connection to user space * @sk socket of connection to user space
*/ */
static void af_alg_data_wakeup(struct sock *sk)
void af_alg_data_wakeup(struct sock *sk)
{ {
struct alg_sock *ask = alg_sk(sk); struct alg_sock *ask = alg_sk(sk);
struct af_alg_ctx *ctx = ask->private; struct af_alg_ctx *ctx = ask->private;
@ -807,7 +798,6 @@ void af_alg_data_wakeup(struct sock *sk)
sk_wake_async(sk, SOCK_WAKE_SPACE, POLL_OUT); sk_wake_async(sk, SOCK_WAKE_SPACE, POLL_OUT);
rcu_read_unlock(); rcu_read_unlock();
} }
EXPORT_SYMBOL_GPL(af_alg_data_wakeup);
/** /**
* af_alg_sendmsg - implementation of sendmsg system call handler * af_alg_sendmsg - implementation of sendmsg system call handler

42
crypto/ahash.c
View File

@ -86,17 +86,17 @@ static int hash_walk_new_entry(struct crypto_hash_walk *walk)
int crypto_hash_walk_done(struct crypto_hash_walk *walk, int err) int crypto_hash_walk_done(struct crypto_hash_walk *walk, int err)
{ {
unsigned int alignmask = walk->alignmask; unsigned int alignmask = walk->alignmask;
unsigned int nbytes = walk->entrylen;
walk->data -= walk->offset; walk->data -= walk->offset;
if (nbytes && walk->offset & alignmask && !err) { if (walk->entrylen && (walk->offset & alignmask) && !err) {
walk->offset = ALIGN(walk->offset, alignmask + 1); unsigned int nbytes;
nbytes = min(nbytes,
((unsigned int)(PAGE_SIZE)) - walk->offset);
walk->entrylen -= nbytes;
walk->offset = ALIGN(walk->offset, alignmask + 1);
nbytes = min(walk->entrylen,
(unsigned int)(PAGE_SIZE - walk->offset));
if (nbytes) { if (nbytes) {
walk->entrylen -= nbytes;
walk->data += walk->offset; walk->data += walk->offset;
return nbytes; return nbytes;
} }
@ -116,7 +116,7 @@ int crypto_hash_walk_done(struct crypto_hash_walk *walk, int err)
if (err) if (err)
return err; return err;
if (nbytes) { if (walk->entrylen) {
walk->offset = 0; walk->offset = 0;
walk->pg++; walk->pg++;
return hash_walk_next(walk); return hash_walk_next(walk);
@ -190,6 +190,21 @@ static int ahash_setkey_unaligned(struct crypto_ahash *tfm, const u8 *key,
return ret; return ret;
} }
static int ahash_nosetkey(struct crypto_ahash *tfm, const u8 *key,
unsigned int keylen)
{
return -ENOSYS;
}
static void ahash_set_needkey(struct crypto_ahash *tfm)
{
const struct hash_alg_common *alg = crypto_hash_alg_common(tfm);
if (tfm->setkey != ahash_nosetkey &&
!(alg->base.cra_flags & CRYPTO_ALG_OPTIONAL_KEY))
crypto_ahash_set_flags(tfm, CRYPTO_TFM_NEED_KEY);
}
int crypto_ahash_setkey(struct crypto_ahash *tfm, const u8 *key, int crypto_ahash_setkey(struct crypto_ahash *tfm, const u8 *key,
unsigned int keylen) unsigned int keylen)
{ {
@ -201,20 +216,16 @@ int crypto_ahash_setkey(struct crypto_ahash *tfm, const u8 *key,
else else
err = tfm->setkey(tfm, key, keylen); err = tfm->setkey(tfm, key, keylen);
if (err) if (unlikely(err)) {
ahash_set_needkey(tfm);
return err; return err;
}
crypto_ahash_clear_flags(tfm, CRYPTO_TFM_NEED_KEY); crypto_ahash_clear_flags(tfm, CRYPTO_TFM_NEED_KEY);
return 0; return 0;
} }
EXPORT_SYMBOL_GPL(crypto_ahash_setkey); EXPORT_SYMBOL_GPL(crypto_ahash_setkey);
static int ahash_nosetkey(struct crypto_ahash *tfm, const u8 *key,
unsigned int keylen)
{
return -ENOSYS;
}
static inline unsigned int ahash_align_buffer_size(unsigned len, static inline unsigned int ahash_align_buffer_size(unsigned len,
unsigned long mask) unsigned long mask)
{ {
@ -489,8 +500,7 @@ static int crypto_ahash_init_tfm(struct crypto_tfm *tfm)
if (alg->setkey) { if (alg->setkey) {
hash->setkey = alg->setkey; hash->setkey = alg->setkey;
if (!(alg->halg.base.cra_flags & CRYPTO_ALG_OPTIONAL_KEY)) ahash_set_needkey(hash);
crypto_ahash_set_flags(hash, CRYPTO_TFM_NEED_KEY);
} }
return 0; return 0;

63
crypto/algapi.c
View File

@ -494,6 +494,24 @@ out:
} }
EXPORT_SYMBOL_GPL(crypto_register_template); EXPORT_SYMBOL_GPL(crypto_register_template);
int crypto_register_templates(struct crypto_template *tmpls, int count)
{
int i, err;
for (i = 0; i < count; i++) {
err = crypto_register_template(&tmpls[i]);
if (err)
goto out;
}
return 0;
out:
for (--i; i >= 0; --i)
crypto_unregister_template(&tmpls[i]);
return err;
}
EXPORT_SYMBOL_GPL(crypto_register_templates);
void crypto_unregister_template(struct crypto_template *tmpl) void crypto_unregister_template(struct crypto_template *tmpl)
{ {
struct crypto_instance *inst; struct crypto_instance *inst;
@ -523,6 +541,15 @@ void crypto_unregister_template(struct crypto_template *tmpl)
} }
EXPORT_SYMBOL_GPL(crypto_unregister_template); EXPORT_SYMBOL_GPL(crypto_unregister_template);
void crypto_unregister_templates(struct crypto_template *tmpls, int count)
{
int i;
for (i = count - 1; i >= 0; --i)
crypto_unregister_template(&tmpls[i]);
}
EXPORT_SYMBOL_GPL(crypto_unregister_templates);
static struct crypto_template *__crypto_lookup_template(const char *name) static struct crypto_template *__crypto_lookup_template(const char *name)
{ {
struct crypto_template *q, *tmpl = NULL; struct crypto_template *q, *tmpl = NULL;
@ -608,6 +635,9 @@ int crypto_init_spawn(struct crypto_spawn *spawn, struct crypto_alg *alg,
{ {
int err = -EAGAIN; int err = -EAGAIN;
if (WARN_ON_ONCE(inst == NULL))
return -EINVAL;
spawn->inst = inst; spawn->inst = inst;
spawn->mask = mask; spawn->mask = mask;
@ -845,8 +875,8 @@ int crypto_inst_setname(struct crypto_instance *inst, const char *name,
} }
EXPORT_SYMBOL_GPL(crypto_inst_setname); EXPORT_SYMBOL_GPL(crypto_inst_setname);
void *crypto_alloc_instance2(const char *name, struct crypto_alg *alg, void *crypto_alloc_instance(const char *name, struct crypto_alg *alg,
unsigned int head) unsigned int head)
{ {
struct crypto_instance *inst; struct crypto_instance *inst;
char *p; char *p;
@ -869,35 +899,6 @@ err_free_inst:
kfree(p); kfree(p);
return ERR_PTR(err); return ERR_PTR(err);
} }
EXPORT_SYMBOL_GPL(crypto_alloc_instance2);
struct crypto_instance *crypto_alloc_instance(const char *name,
struct crypto_alg *alg)
{
struct crypto_instance *inst;
struct crypto_spawn *spawn;
int err;
inst = crypto_alloc_instance2(name, alg, 0);
if (IS_ERR(inst))
goto out;
spawn = crypto_instance_ctx(inst);
err = crypto_init_spawn(spawn, alg, inst,
CRYPTO_ALG_TYPE_MASK | CRYPTO_ALG_ASYNC);
if (err)
goto err_free_inst;
return inst;
err_free_inst:
kfree(inst);
inst = ERR_PTR(err);
out:
return inst;
}
EXPORT_SYMBOL_GPL(crypto_alloc_instance); EXPORT_SYMBOL_GPL(crypto_alloc_instance);
void crypto_init_queue(struct crypto_queue *queue, unsigned int max_qlen) void crypto_init_queue(struct crypto_queue *queue, unsigned int max_qlen)

87
crypto/arc4.c
View File

@ -12,14 +12,11 @@
* *
*/ */
#include <linux/module.h>
#include <linux/init.h>
#include <linux/crypto.h>
#include <crypto/algapi.h> #include <crypto/algapi.h>
#include <crypto/arc4.h>
#define ARC4_MIN_KEY_SIZE 1 #include <crypto/internal/skcipher.h>
#define ARC4_MAX_KEY_SIZE 256 #include <linux/init.h>
#define ARC4_BLOCK_SIZE 1 #include <linux/module.h>
struct arc4_ctx { struct arc4_ctx {
u32 S[256]; u32 S[256];
@ -50,6 +47,12 @@ static int arc4_set_key(struct crypto_tfm *tfm, const u8 *in_key,
return 0; return 0;
} }
static int arc4_set_key_skcipher(struct crypto_skcipher *tfm, const u8 *in_key,
unsigned int key_len)
{
return arc4_set_key(&tfm->base, in_key, key_len);
}
static void arc4_crypt(struct arc4_ctx *ctx, u8 *out, const u8 *in, static void arc4_crypt(struct arc4_ctx *ctx, u8 *out, const u8 *in,
unsigned int len) unsigned int len)
{ {
@ -92,30 +95,25 @@ static void arc4_crypt_one(struct crypto_tfm *tfm, u8 *out, const u8 *in)
arc4_crypt(crypto_tfm_ctx(tfm), out, in, 1); arc4_crypt(crypto_tfm_ctx(tfm), out, in, 1);
} }
static int ecb_arc4_crypt(struct blkcipher_desc *desc, struct scatterlist *dst, static int ecb_arc4_crypt(struct skcipher_request *req)
struct scatterlist *src, unsigned int nbytes)
{ {
struct arc4_ctx *ctx = crypto_blkcipher_ctx(desc->tfm); struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct blkcipher_walk walk; struct arc4_ctx *ctx = crypto_skcipher_ctx(tfm);
struct skcipher_walk walk;
int err; int err;
blkcipher_walk_init(&walk, dst, src, nbytes); err = skcipher_walk_virt(&walk, req, false);
err = blkcipher_walk_virt(desc, &walk);
while (walk.nbytes > 0) { while (walk.nbytes > 0) {
u8 *wsrc = walk.src.virt.addr; arc4_crypt(ctx, walk.dst.virt.addr, walk.src.virt.addr,
u8 *wdst = walk.dst.virt.addr; walk.nbytes);
err = skcipher_walk_done(&walk, 0);
arc4_crypt(ctx, wdst, wsrc, walk.nbytes);
err = blkcipher_walk_done(desc, &walk, 0);
} }
return err; return err;
} }
static struct crypto_alg arc4_algs[2] = { { static struct crypto_alg arc4_cipher = {
.cra_name = "arc4", .cra_name = "arc4",
.cra_flags = CRYPTO_ALG_TYPE_CIPHER, .cra_flags = CRYPTO_ALG_TYPE_CIPHER,
.cra_blocksize = ARC4_BLOCK_SIZE, .cra_blocksize = ARC4_BLOCK_SIZE,
@ -130,34 +128,39 @@ static struct crypto_alg arc4_algs[2] = { {
.cia_decrypt = arc4_crypt_one, .cia_decrypt = arc4_crypt_one,
}, },
}, },
}, { };
.cra_name = "ecb(arc4)",
.cra_priority = 100, static struct skcipher_alg arc4_skcipher = {
.cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER, .base.cra_name = "ecb(arc4)",
.cra_blocksize = ARC4_BLOCK_SIZE, .base.cra_priority = 100,
.cra_ctxsize = sizeof(struct arc4_ctx), .base.cra_blocksize = ARC4_BLOCK_SIZE,
.cra_alignmask = 0, .base.cra_ctxsize = sizeof(struct arc4_ctx),
.cra_type = &crypto_blkcipher_type, .base.cra_module = THIS_MODULE,
.cra_module = THIS_MODULE, .min_keysize = ARC4_MIN_KEY_SIZE,
.cra_u = { .max_keysize = ARC4_MAX_KEY_SIZE,
.blkcipher = { .setkey = arc4_set_key_skcipher,
.min_keysize = ARC4_MIN_KEY_SIZE, .encrypt = ecb_arc4_crypt,
.max_keysize = ARC4_MAX_KEY_SIZE, .decrypt = ecb_arc4_crypt,
.setkey = arc4_set_key, };
.encrypt = ecb_arc4_crypt,
.decrypt = ecb_arc4_crypt,
},
},
} };
static int __init arc4_init(void) static int __init arc4_init(void)
{ {
return crypto_register_algs(arc4_algs, ARRAY_SIZE(arc4_algs)); int err;
err = crypto_register_alg(&arc4_cipher);
if (err)
return err;
err = crypto_register_skcipher(&arc4_skcipher);
if (err)
crypto_unregister_alg(&arc4_cipher);
return err;
} }
static void __exit arc4_exit(void) static void __exit arc4_exit(void)
{ {
crypto_unregister_algs(arc4_algs, ARRAY_SIZE(arc4_algs)); crypto_unregister_alg(&arc4_cipher);
crypto_unregister_skcipher(&arc4_skcipher);
} }
module_init(arc4_init); module_init(arc4_init);

131
crypto/cbc.c
View File

@ -18,34 +18,11 @@
#include <linux/kernel.h> #include <linux/kernel.h>
#include <linux/log2.h> #include <linux/log2.h>
#include <linux/module.h> #include <linux/module.h>
#include <linux/slab.h>
struct crypto_cbc_ctx {
struct crypto_cipher *child;
};
static int crypto_cbc_setkey(struct crypto_skcipher *parent, const u8 *key,
unsigned int keylen)
{
struct crypto_cbc_ctx *ctx = crypto_skcipher_ctx(parent);
struct crypto_cipher *child = ctx->child;
int err;
crypto_cipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
crypto_cipher_set_flags(child, crypto_skcipher_get_flags(parent) &
CRYPTO_TFM_REQ_MASK);
err = crypto_cipher_setkey(child, key, keylen);
crypto_skcipher_set_flags(parent, crypto_cipher_get_flags(child) &
CRYPTO_TFM_RES_MASK);
return err;
}
static inline void crypto_cbc_encrypt_one(struct crypto_skcipher *tfm, static inline void crypto_cbc_encrypt_one(struct crypto_skcipher *tfm,
const u8 *src, u8 *dst) const u8 *src, u8 *dst)
{ {
struct crypto_cbc_ctx *ctx = crypto_skcipher_ctx(tfm); crypto_cipher_encrypt_one(skcipher_cipher_simple(tfm), dst, src);
crypto_cipher_encrypt_one(ctx->child, dst, src);
} }
static int crypto_cbc_encrypt(struct skcipher_request *req) static int crypto_cbc_encrypt(struct skcipher_request *req)
@ -56,9 +33,7 @@ static int crypto_cbc_encrypt(struct skcipher_request *req)
static inline void crypto_cbc_decrypt_one(struct crypto_skcipher *tfm, static inline void crypto_cbc_decrypt_one(struct crypto_skcipher *tfm,
const u8 *src, u8 *dst) const u8 *src, u8 *dst)
{ {
struct crypto_cbc_ctx *ctx = crypto_skcipher_ctx(tfm); crypto_cipher_decrypt_one(skcipher_cipher_simple(tfm), dst, src);
crypto_cipher_decrypt_one(ctx->child, dst, src);
} }
static int crypto_cbc_decrypt(struct skcipher_request *req) static int crypto_cbc_decrypt(struct skcipher_request *req)
@ -78,113 +53,33 @@ static int crypto_cbc_decrypt(struct skcipher_request *req)
return err; return err;
} }
static int crypto_cbc_init_tfm(struct crypto_skcipher *tfm)
{
struct skcipher_instance *inst = skcipher_alg_instance(tfm);
struct crypto_spawn *spawn = skcipher_instance_ctx(inst);
struct crypto_cbc_ctx *ctx = crypto_skcipher_ctx(tfm);
struct crypto_cipher *cipher;
cipher = crypto_spawn_cipher(spawn);
if (IS_ERR(cipher))
return PTR_ERR(cipher);
ctx->child = cipher;
return 0;
}
static void crypto_cbc_exit_tfm(struct crypto_skcipher *tfm)
{
struct crypto_cbc_ctx *ctx = crypto_skcipher_ctx(tfm);
crypto_free_cipher(ctx->child);
}
static void crypto_cbc_free(struct skcipher_instance *inst)
{
crypto_drop_skcipher(skcipher_instance_ctx(inst));
kfree(inst);
}
static int crypto_cbc_create(struct crypto_template *tmpl, struct rtattr **tb) static int crypto_cbc_create(struct crypto_template *tmpl, struct rtattr **tb)
{ {
struct skcipher_instance *inst; struct skcipher_instance *inst;
struct crypto_attr_type *algt;
struct crypto_spawn *spawn;
struct crypto_alg *alg; struct crypto_alg *alg;
u32 mask;
int err; int err;
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_SKCIPHER); inst = skcipher_alloc_instance_simple(tmpl, tb, &alg);
if (err) if (IS_ERR(inst))
return err; return PTR_ERR(inst);
inst = kzalloc(sizeof(*inst) + sizeof(*spawn), GFP_KERNEL);
if (!inst)
return -ENOMEM;
algt = crypto_get_attr_type(tb);
err = PTR_ERR(algt);
if (IS_ERR(algt))
goto err_free_inst;
mask = CRYPTO_ALG_TYPE_MASK |
crypto_requires_off(algt->type, algt->mask,
CRYPTO_ALG_NEED_FALLBACK);
alg = crypto_get_attr_alg(tb, CRYPTO_ALG_TYPE_CIPHER, mask);
err = PTR_ERR(alg);
if (IS_ERR(alg))
goto err_free_inst;
spawn = skcipher_instance_ctx(inst);
err = crypto_init_spawn(spawn, alg, skcipher_crypto_instance(inst),
CRYPTO_ALG_TYPE_MASK);
if (err)
goto err_put_alg;
err = crypto_inst_setname(skcipher_crypto_instance(inst), "cbc", alg);
if (err)
goto err_drop_spawn;
err = -EINVAL; err = -EINVAL;
if (!is_power_of_2(alg->cra_blocksize)) if (!is_power_of_2(alg->cra_blocksize))
goto err_drop_spawn; goto out_free_inst;
inst->alg.base.cra_priority = alg->cra_priority;
inst->alg.base.cra_blocksize = alg->cra_blocksize;
inst->alg.base.cra_alignmask = alg->cra_alignmask;
inst->alg.ivsize = alg->cra_blocksize;
inst->alg.min_keysize = alg->cra_cipher.cia_min_keysize;
inst->alg.max_keysize = alg->cra_cipher.cia_max_keysize;
inst->alg.base.cra_ctxsize = sizeof(struct crypto_cbc_ctx);
inst->alg.init = crypto_cbc_init_tfm;
inst->alg.exit = crypto_cbc_exit_tfm;
inst->alg.setkey = crypto_cbc_setkey;
inst->alg.encrypt = crypto_cbc_encrypt; inst->alg.encrypt = crypto_cbc_encrypt;
inst->alg.decrypt = crypto_cbc_decrypt; inst->alg.decrypt = crypto_cbc_decrypt;
inst->free = crypto_cbc_free;
err = skcipher_register_instance(tmpl, inst); err = skcipher_register_instance(tmpl, inst);
if (err) if (err)
goto err_drop_spawn; goto out_free_inst;
crypto_mod_put(alg); goto out_put_alg;
out: out_free_inst:
inst->free(inst);
out_put_alg:
crypto_mod_put(alg);
return err; return err;
err_drop_spawn:
crypto_drop_spawn(spawn);
err_put_alg:
crypto_mod_put(alg);
err_free_inst:
kfree(inst);
goto out;
} }
static struct crypto_template crypto_cbc_tmpl = { static struct crypto_template crypto_cbc_tmpl = {
@ -207,5 +102,5 @@ module_init(crypto_cbc_module_init);
module_exit(crypto_cbc_module_exit); module_exit(crypto_cbc_module_exit);
MODULE_LICENSE("GPL"); MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("CBC block cipher algorithm"); MODULE_DESCRIPTION("CBC block cipher mode of operation");
MODULE_ALIAS_CRYPTO("cbc"); MODULE_ALIAS_CRYPTO("cbc");

78
crypto/ccm.c
View File

@ -589,12 +589,6 @@ static int crypto_ccm_create(struct crypto_template *tmpl, struct rtattr **tb)
mac_name); mac_name);
} }
static struct crypto_template crypto_ccm_tmpl = {
.name = "ccm",
.create = crypto_ccm_create,
.module = THIS_MODULE,
};
static int crypto_ccm_base_create(struct crypto_template *tmpl, static int crypto_ccm_base_create(struct crypto_template *tmpl,
struct rtattr **tb) struct rtattr **tb)
{ {
@ -618,12 +612,6 @@ static int crypto_ccm_base_create(struct crypto_template *tmpl,
cipher_name); cipher_name);
} }
static struct crypto_template crypto_ccm_base_tmpl = {
.name = "ccm_base",
.create = crypto_ccm_base_create,
.module = THIS_MODULE,
};
static int crypto_rfc4309_setkey(struct crypto_aead *parent, const u8 *key, static int crypto_rfc4309_setkey(struct crypto_aead *parent, const u8 *key,
unsigned int keylen) unsigned int keylen)
{ {
@ -854,12 +842,6 @@ out_free_inst:
goto out; goto out;
} }
static struct crypto_template crypto_rfc4309_tmpl = {
.name = "rfc4309",
.create = crypto_rfc4309_create,
.module = THIS_MODULE,
};
static int crypto_cbcmac_digest_setkey(struct crypto_shash *parent, static int crypto_cbcmac_digest_setkey(struct crypto_shash *parent,
const u8 *inkey, unsigned int keylen) const u8 *inkey, unsigned int keylen)
{ {
@ -999,51 +981,37 @@ out_put_alg:
return err; return err;
} }
static struct crypto_template crypto_cbcmac_tmpl = { static struct crypto_template crypto_ccm_tmpls[] = {
.name = "cbcmac", {
.create = cbcmac_create, .name = "cbcmac",
.free = shash_free_instance, .create = cbcmac_create,
.module = THIS_MODULE, .free = shash_free_instance,
.module = THIS_MODULE,
}, {
.name = "ccm_base",
.create = crypto_ccm_base_create,
.module = THIS_MODULE,
}, {
.name = "ccm",
.create = crypto_ccm_create,
.module = THIS_MODULE,
}, {
.name = "rfc4309",
.create = crypto_rfc4309_create,
.module = THIS_MODULE,
},
}; };
static int __init crypto_ccm_module_init(void) static int __init crypto_ccm_module_init(void)
{ {
int err; return crypto_register_templates(crypto_ccm_tmpls,
ARRAY_SIZE(crypto_ccm_tmpls));
err = crypto_register_template(&crypto_cbcmac_tmpl);
if (err)
goto out;
err = crypto_register_template(&crypto_ccm_base_tmpl);
if (err)
goto out_undo_cbcmac;
err = crypto_register_template(&crypto_ccm_tmpl);
if (err)
goto out_undo_base;
err = crypto_register_template(&crypto_rfc4309_tmpl);
if (err)
goto out_undo_ccm;
out:
return err;
out_undo_ccm:
crypto_unregister_template(&crypto_ccm_tmpl);
out_undo_base:
crypto_unregister_template(&crypto_ccm_base_tmpl);
out_undo_cbcmac:
crypto_register_template(&crypto_cbcmac_tmpl);
goto out;
} }
static void __exit crypto_ccm_module_exit(void) static void __exit crypto_ccm_module_exit(void)
{ {
crypto_unregister_template(&crypto_rfc4309_tmpl); crypto_unregister_templates(crypto_ccm_tmpls,
crypto_unregister_template(&crypto_ccm_tmpl); ARRAY_SIZE(crypto_ccm_tmpls));
crypto_unregister_template(&crypto_ccm_base_tmpl);
crypto_unregister_template(&crypto_cbcmac_tmpl);
} }
module_init(crypto_ccm_module_init); module_init(crypto_ccm_module_init);

139
crypto/cfb.c
View File

@ -25,28 +25,17 @@
#include <linux/init.h> #include <linux/init.h>
#include <linux/kernel.h> #include <linux/kernel.h>
#include <linux/module.h> #include <linux/module.h>
#include <linux/slab.h>
#include <linux/string.h> #include <linux/string.h>
#include <linux/types.h>
struct crypto_cfb_ctx {
struct crypto_cipher *child;
};
static unsigned int crypto_cfb_bsize(struct crypto_skcipher *tfm) static unsigned int crypto_cfb_bsize(struct crypto_skcipher *tfm)
{ {
struct crypto_cfb_ctx *ctx = crypto_skcipher_ctx(tfm); return crypto_cipher_blocksize(skcipher_cipher_simple(tfm));
struct crypto_cipher *child = ctx->child;
return crypto_cipher_blocksize(child);
} }
static void crypto_cfb_encrypt_one(struct crypto_skcipher *tfm, static void crypto_cfb_encrypt_one(struct crypto_skcipher *tfm,
const u8 *src, u8 *dst) const u8 *src, u8 *dst)
{ {
struct crypto_cfb_ctx *ctx = crypto_skcipher_ctx(tfm); crypto_cipher_encrypt_one(skcipher_cipher_simple(tfm), dst, src);
crypto_cipher_encrypt_one(ctx->child, dst, src);
} }
/* final encrypt and decrypt is the same */ /* final encrypt and decrypt is the same */
@ -77,12 +66,14 @@ static int crypto_cfb_encrypt_segment(struct skcipher_walk *walk,
do { do {
crypto_cfb_encrypt_one(tfm, iv, dst); crypto_cfb_encrypt_one(tfm, iv, dst);
crypto_xor(dst, src, bsize); crypto_xor(dst, src, bsize);
memcpy(iv, dst, bsize); iv = dst;
src += bsize; src += bsize;
dst += bsize; dst += bsize;
} while ((nbytes -= bsize) >= bsize); } while ((nbytes -= bsize) >= bsize);
memcpy(walk->iv, iv, bsize);
return nbytes; return nbytes;
} }
@ -162,7 +153,7 @@ static int crypto_cfb_decrypt_inplace(struct skcipher_walk *walk,
const unsigned int bsize = crypto_cfb_bsize(tfm); const unsigned int bsize = crypto_cfb_bsize(tfm);
unsigned int nbytes = walk->nbytes; unsigned int nbytes = walk->nbytes;
u8 *src = walk->src.virt.addr; u8 *src = walk->src.virt.addr;
u8 *iv = walk->iv; u8 * const iv = walk->iv;
u8 tmp[MAX_CIPHER_BLOCKSIZE]; u8 tmp[MAX_CIPHER_BLOCKSIZE];
do { do {
@ -172,8 +163,6 @@ static int crypto_cfb_decrypt_inplace(struct skcipher_walk *walk,
src += bsize; src += bsize;
} while ((nbytes -= bsize) >= bsize); } while ((nbytes -= bsize) >= bsize);
memcpy(walk->iv, iv, bsize);
return nbytes; return nbytes;
} }
@ -186,22 +175,6 @@ static int crypto_cfb_decrypt_blocks(struct skcipher_walk *walk,
return crypto_cfb_decrypt_segment(walk, tfm); return crypto_cfb_decrypt_segment(walk, tfm);
} }
static int crypto_cfb_setkey(struct crypto_skcipher *parent, const u8 *key,
unsigned int keylen)
{
struct crypto_cfb_ctx *ctx = crypto_skcipher_ctx(parent);
struct crypto_cipher *child = ctx->child;
int err;
crypto_cipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
crypto_cipher_set_flags(child, crypto_skcipher_get_flags(parent) &
CRYPTO_TFM_REQ_MASK);
err = crypto_cipher_setkey(child, key, keylen);
crypto_skcipher_set_flags(parent, crypto_cipher_get_flags(child) &
CRYPTO_TFM_RES_MASK);
return err;
}
static int crypto_cfb_decrypt(struct skcipher_request *req) static int crypto_cfb_decrypt(struct skcipher_request *req)
{ {
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
@ -224,110 +197,34 @@ static int crypto_cfb_decrypt(struct skcipher_request *req)
return err; return err;
} }
static int crypto_cfb_init_tfm(struct crypto_skcipher *tfm)
{
struct skcipher_instance *inst = skcipher_alg_instance(tfm);
struct crypto_spawn *spawn = skcipher_instance_ctx(inst);
struct crypto_cfb_ctx *ctx = crypto_skcipher_ctx(tfm);
struct crypto_cipher *cipher;
cipher = crypto_spawn_cipher(spawn);
if (IS_ERR(cipher))
return PTR_ERR(cipher);
ctx->child = cipher;
return 0;
}
static void crypto_cfb_exit_tfm(struct crypto_skcipher *tfm)
{
struct crypto_cfb_ctx *ctx = crypto_skcipher_ctx(tfm);
crypto_free_cipher(ctx->child);
}
static void crypto_cfb_free(struct skcipher_instance *inst)
{
crypto_drop_skcipher(skcipher_instance_ctx(inst));
kfree(inst);
}
static int crypto_cfb_create(struct crypto_template *tmpl, struct rtattr **tb) static int crypto_cfb_create(struct crypto_template *tmpl, struct rtattr **tb)
{ {
struct skcipher_instance *inst; struct skcipher_instance *inst;
struct crypto_attr_type *algt;
struct crypto_spawn *spawn;
struct crypto_alg *alg; struct crypto_alg *alg;
u32 mask;
int err; int err;
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_SKCIPHER); inst = skcipher_alloc_instance_simple(tmpl, tb, &alg);
if (err) if (IS_ERR(inst))
return err; return PTR_ERR(inst);
inst = kzalloc(sizeof(*inst) + sizeof(*spawn), GFP_KERNEL); /* CFB mode is a stream cipher. */
if (!inst)
return -ENOMEM;
algt = crypto_get_attr_type(tb);
err = PTR_ERR(algt);
if (IS_ERR(algt))
goto err_free_inst;
mask = CRYPTO_ALG_TYPE_MASK |
crypto_requires_off(algt->type, algt->mask,
CRYPTO_ALG_NEED_FALLBACK);
alg = crypto_get_attr_alg(tb, CRYPTO_ALG_TYPE_CIPHER, mask);
err = PTR_ERR(alg);
if (IS_ERR(alg))
goto err_free_inst;
spawn = skcipher_instance_ctx(inst);
err = crypto_init_spawn(spawn, alg, skcipher_crypto_instance(inst),
CRYPTO_ALG_TYPE_MASK);
if (err)
goto err_put_alg;
err = crypto_inst_setname(skcipher_crypto_instance(inst), "cfb", alg);
if (err)
goto err_drop_spawn;
inst->alg.base.cra_priority = alg->cra_priority;
/* we're a stream cipher independend of the crypto cra_blocksize */
inst->alg.base.cra_blocksize = 1; inst->alg.base.cra_blocksize = 1;
inst->alg.base.cra_alignmask = alg->cra_alignmask;
inst->alg.ivsize = alg->cra_blocksize; /*
inst->alg.min_keysize = alg->cra_cipher.cia_min_keysize; * To simplify the implementation, configure the skcipher walk to only
inst->alg.max_keysize = alg->cra_cipher.cia_max_keysize; * give a partial block at the very end, never earlier.
*/
inst->alg.chunksize = alg->cra_blocksize;
inst->alg.base.cra_ctxsize = sizeof(struct crypto_cfb_ctx);
inst->alg.init = crypto_cfb_init_tfm;
inst->alg.exit = crypto_cfb_exit_tfm;
inst->alg.setkey = crypto_cfb_setkey;
inst->alg.encrypt = crypto_cfb_encrypt; inst->alg.encrypt = crypto_cfb_encrypt;
inst->alg.decrypt = crypto_cfb_decrypt; inst->alg.decrypt = crypto_cfb_decrypt;
inst->free = crypto_cfb_free;
err = skcipher_register_instance(tmpl, inst); err = skcipher_register_instance(tmpl, inst);
if (err) if (err)
goto err_drop_spawn; inst->free(inst);
crypto_mod_put(alg);
out: crypto_mod_put(alg);
return err; return err;
err_drop_spawn:
crypto_drop_spawn(spawn);
err_put_alg:
crypto_mod_put(alg);
err_free_inst:
kfree(inst);
goto out;
} }
static struct crypto_template crypto_cfb_tmpl = { static struct crypto_template crypto_cfb_tmpl = {
@ -350,5 +247,5 @@ module_init(crypto_cfb_module_init);
module_exit(crypto_cfb_module_exit); module_exit(crypto_cfb_module_exit);
MODULE_LICENSE("GPL"); MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("CFB block cipher algorithm"); MODULE_DESCRIPTION("CFB block cipher mode of operation");
MODULE_ALIAS_CRYPTO("cfb"); MODULE_ALIAS_CRYPTO("cfb");

37
crypto/chacha20poly1305.c
View File

@ -701,37 +701,28 @@ static int rfc7539esp_create(struct crypto_template *tmpl, struct rtattr **tb)
return chachapoly_create(tmpl, tb, "rfc7539esp", 8); return chachapoly_create(tmpl, tb, "rfc7539esp", 8);
} }
static struct crypto_template rfc7539_tmpl = { static struct crypto_template rfc7539_tmpls[] = {
.name = "rfc7539", {
.create = rfc7539_create, .name = "rfc7539",
.module = THIS_MODULE, .create = rfc7539_create,
}; .module = THIS_MODULE,
}, {
static struct crypto_template rfc7539esp_tmpl = { .name = "rfc7539esp",
.name = "rfc7539esp", .create = rfc7539esp_create,
.create = rfc7539esp_create, .module = THIS_MODULE,
.module = THIS_MODULE, },
}; };
static int __init chacha20poly1305_module_init(void) static int __init chacha20poly1305_module_init(void)
{ {
int err; return crypto_register_templates(rfc7539_tmpls,
ARRAY_SIZE(rfc7539_tmpls));
err = crypto_register_template(&rfc7539_tmpl);
if (err)
return err;
err = crypto_register_template(&rfc7539esp_tmpl);
if (err)
crypto_unregister_template(&rfc7539_tmpl);
return err;
} }
static void __exit chacha20poly1305_module_exit(void) static void __exit chacha20poly1305_module_exit(void)
{ {
crypto_unregister_template(&rfc7539esp_tmpl); crypto_unregister_templates(rfc7539_tmpls,
crypto_unregister_template(&rfc7539_tmpl); ARRAY_SIZE(rfc7539_tmpls));
} }
module_init(chacha20poly1305_module_init); module_init(chacha20poly1305_module_init);

57
crypto/crypto_null.c
View File

@ -65,6 +65,10 @@ static int null_hash_setkey(struct crypto_shash *tfm, const u8 *key,
unsigned int keylen) unsigned int keylen)
{ return 0; } { return 0; }
static int null_skcipher_setkey(struct crypto_skcipher *tfm, const u8 *key,
unsigned int keylen)
{ return 0; }
static int null_setkey(struct crypto_tfm *tfm, const u8 *key, static int null_setkey(struct crypto_tfm *tfm, const u8 *key,
unsigned int keylen) unsigned int keylen)
{ return 0; } { return 0; }
@ -74,21 +78,18 @@ static void null_crypt(struct crypto_tfm *tfm, u8 *dst, const u8 *src)
memcpy(dst, src, NULL_BLOCK_SIZE); memcpy(dst, src, NULL_BLOCK_SIZE);
} }
static int skcipher_null_crypt(struct blkcipher_desc *desc, static int null_skcipher_crypt(struct skcipher_request *req)
struct scatterlist *dst,
struct scatterlist *src, unsigned int nbytes)
{ {
struct blkcipher_walk walk; struct skcipher_walk walk;
int err; int err;
blkcipher_walk_init(&walk, dst, src, nbytes); err = skcipher_walk_virt(&walk, req, false);
err = blkcipher_walk_virt(desc, &walk);
while (walk.nbytes) { while (walk.nbytes) {
if (walk.src.virt.addr != walk.dst.virt.addr) if (walk.src.virt.addr != walk.dst.virt.addr)
memcpy(walk.dst.virt.addr, walk.src.virt.addr, memcpy(walk.dst.virt.addr, walk.src.virt.addr,
walk.nbytes); walk.nbytes);
err = blkcipher_walk_done(desc, &walk, 0); err = skcipher_walk_done(&walk, 0);
} }
return err; return err;
@ -109,7 +110,22 @@ static struct shash_alg digest_null = {
} }
}; };
static struct crypto_alg null_algs[3] = { { static struct skcipher_alg skcipher_null = {
.base.cra_name = "ecb(cipher_null)",
.base.cra_driver_name = "ecb-cipher_null",
.base.cra_priority = 100,
.base.cra_blocksize = NULL_BLOCK_SIZE,
.base.cra_ctxsize = 0,
.base.cra_module = THIS_MODULE,
.min_keysize = NULL_KEY_SIZE,
.max_keysize = NULL_KEY_SIZE,
.ivsize = NULL_IV_SIZE,
.setkey = null_skcipher_setkey,
.encrypt = null_skcipher_crypt,
.decrypt = null_skcipher_crypt,
};
static struct crypto_alg null_algs[] = { {
.cra_name = "cipher_null", .cra_name = "cipher_null",
.cra_flags = CRYPTO_ALG_TYPE_CIPHER, .cra_flags = CRYPTO_ALG_TYPE_CIPHER,
.cra_blocksize = NULL_BLOCK_SIZE, .cra_blocksize = NULL_BLOCK_SIZE,
@ -121,22 +137,6 @@ static struct crypto_alg null_algs[3] = { {
.cia_setkey = null_setkey, .cia_setkey = null_setkey,
.cia_encrypt = null_crypt, .cia_encrypt = null_crypt,
.cia_decrypt = null_crypt } } .cia_decrypt = null_crypt } }
}, {
.cra_name = "ecb(cipher_null)",
.cra_driver_name = "ecb-cipher_null",
.cra_priority = 100,
.cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER,
.cra_blocksize = NULL_BLOCK_SIZE,
.cra_type = &crypto_blkcipher_type,
.cra_ctxsize = 0,
.cra_module = THIS_MODULE,
.cra_u = { .blkcipher = {
.min_keysize = NULL_KEY_SIZE,
.max_keysize = NULL_KEY_SIZE,
.ivsize = NULL_IV_SIZE,
.setkey = null_setkey,
.encrypt = skcipher_null_crypt,
.decrypt = skcipher_null_crypt } }
}, { }, {
.cra_name = "compress_null", .cra_name = "compress_null",
.cra_flags = CRYPTO_ALG_TYPE_COMPRESS, .cra_flags = CRYPTO_ALG_TYPE_COMPRESS,
@ -199,8 +199,14 @@ static int __init crypto_null_mod_init(void)
if (ret < 0) if (ret < 0)
goto out_unregister_algs; goto out_unregister_algs;
ret = crypto_register_skcipher(&skcipher_null);
if (ret < 0)
goto out_unregister_shash;
return 0; return 0;
out_unregister_shash:
crypto_unregister_shash(&digest_null);
out_unregister_algs: out_unregister_algs:
crypto_unregister_algs(null_algs, ARRAY_SIZE(null_algs)); crypto_unregister_algs(null_algs, ARRAY_SIZE(null_algs));
out: out:
@ -209,8 +215,9 @@ out:
static void __exit crypto_null_mod_fini(void) static void __exit crypto_null_mod_fini(void)
{ {
crypto_unregister_shash(&digest_null);
crypto_unregister_algs(null_algs, ARRAY_SIZE(null_algs)); crypto_unregister_algs(null_algs, ARRAY_SIZE(null_algs));
crypto_unregister_shash(&digest_null);
crypto_unregister_skcipher(&skcipher_null);
} }
module_init(crypto_null_mod_init); module_init(crypto_null_mod_init);

4
crypto/crypto_user_stat.c
View File

@ -20,10 +20,6 @@
#define null_terminated(x) (strnlen(x, sizeof(x)) < sizeof(x)) #define null_terminated(x) (strnlen(x, sizeof(x)) < sizeof(x))
static DEFINE_MUTEX(crypto_cfg_mutex);
extern struct sock *crypto_nlsk;
struct crypto_dump_info { struct crypto_dump_info {
struct sk_buff *in_skb; struct sk_buff *in_skb;
struct sk_buff *out_skb; struct sk_buff *out_skb;

200
crypto/ctr.c
View File

@ -17,14 +17,8 @@
#include <linux/init.h> #include <linux/init.h>
#include <linux/kernel.h> #include <linux/kernel.h>
#include <linux/module.h> #include <linux/module.h>
#include <linux/random.h>
#include <linux/scatterlist.h>
#include <linux/slab.h> #include <linux/slab.h>
struct crypto_ctr_ctx {
struct crypto_cipher *child;
};
struct crypto_rfc3686_ctx { struct crypto_rfc3686_ctx {
struct crypto_skcipher *child; struct crypto_skcipher *child;
u8 nonce[CTR_RFC3686_NONCE_SIZE]; u8 nonce[CTR_RFC3686_NONCE_SIZE];
@ -35,24 +29,7 @@ struct crypto_rfc3686_req_ctx {
struct skcipher_request subreq CRYPTO_MINALIGN_ATTR; struct skcipher_request subreq CRYPTO_MINALIGN_ATTR;
}; };
static int crypto_ctr_setkey(struct crypto_tfm *parent, const u8 *key, static void crypto_ctr_crypt_final(struct skcipher_walk *walk,
unsigned int keylen)
{
struct crypto_ctr_ctx *ctx = crypto_tfm_ctx(parent);
struct crypto_cipher *child = ctx->child;
int err;
crypto_cipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
crypto_cipher_set_flags(child, crypto_tfm_get_flags(parent) &
CRYPTO_TFM_REQ_MASK);
err = crypto_cipher_setkey(child, key, keylen);
crypto_tfm_set_flags(parent, crypto_cipher_get_flags(child) &
CRYPTO_TFM_RES_MASK);
return err;
}
static void crypto_ctr_crypt_final(struct blkcipher_walk *walk,
struct crypto_cipher *tfm) struct crypto_cipher *tfm)
{ {
unsigned int bsize = crypto_cipher_blocksize(tfm); unsigned int bsize = crypto_cipher_blocksize(tfm);
@ -70,7 +47,7 @@ static void crypto_ctr_crypt_final(struct blkcipher_walk *walk,
crypto_inc(ctrblk, bsize); crypto_inc(ctrblk, bsize);
} }
static int crypto_ctr_crypt_segment(struct blkcipher_walk *walk, static int crypto_ctr_crypt_segment(struct skcipher_walk *walk,
struct crypto_cipher *tfm) struct crypto_cipher *tfm)
{ {
void (*fn)(struct crypto_tfm *, u8 *, const u8 *) = void (*fn)(struct crypto_tfm *, u8 *, const u8 *) =
@ -96,7 +73,7 @@ static int crypto_ctr_crypt_segment(struct blkcipher_walk *walk,
return nbytes; return nbytes;
} }
static int crypto_ctr_crypt_inplace(struct blkcipher_walk *walk, static int crypto_ctr_crypt_inplace(struct skcipher_walk *walk,
struct crypto_cipher *tfm) struct crypto_cipher *tfm)
{ {
void (*fn)(struct crypto_tfm *, u8 *, const u8 *) = void (*fn)(struct crypto_tfm *, u8 *, const u8 *) =
@ -123,138 +100,77 @@ static int crypto_ctr_crypt_inplace(struct blkcipher_walk *walk,
return nbytes; return nbytes;
} }
static int crypto_ctr_crypt(struct blkcipher_desc *desc, static int crypto_ctr_crypt(struct skcipher_request *req)
struct scatterlist *dst, struct scatterlist *src,
unsigned int nbytes)
{ {
struct blkcipher_walk walk; struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct crypto_blkcipher *tfm = desc->tfm; struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
struct crypto_ctr_ctx *ctx = crypto_blkcipher_ctx(tfm); const unsigned int bsize = crypto_cipher_blocksize(cipher);
struct crypto_cipher *child = ctx->child; struct skcipher_walk walk;
unsigned int bsize = crypto_cipher_blocksize(child); unsigned int nbytes;
int err; int err;
blkcipher_walk_init(&walk, dst, src, nbytes); err = skcipher_walk_virt(&walk, req, false);
err = blkcipher_walk_virt_block(desc, &walk, bsize);
while (walk.nbytes >= bsize) { while (walk.nbytes >= bsize) {
if (walk.src.virt.addr == walk.dst.virt.addr) if (walk.src.virt.addr == walk.dst.virt.addr)
nbytes = crypto_ctr_crypt_inplace(&walk, child); nbytes = crypto_ctr_crypt_inplace(&walk, cipher);
else else
nbytes = crypto_ctr_crypt_segment(&walk, child); nbytes = crypto_ctr_crypt_segment(&walk, cipher);
err = blkcipher_walk_done(desc, &walk, nbytes); err = skcipher_walk_done(&walk, nbytes);
} }
if (walk.nbytes) { if (walk.nbytes) {
crypto_ctr_crypt_final(&walk, child); crypto_ctr_crypt_final(&walk, cipher);
err = blkcipher_walk_done(desc, &walk, 0); err = skcipher_walk_done(&walk, 0);
} }
return err; return err;
} }
static int crypto_ctr_init_tfm(struct crypto_tfm *tfm) static int crypto_ctr_create(struct crypto_template *tmpl, struct rtattr **tb)
{ {
struct crypto_instance *inst = (void *)tfm->__crt_alg; struct skcipher_instance *inst;
struct crypto_spawn *spawn = crypto_instance_ctx(inst);
struct crypto_ctr_ctx *ctx = crypto_tfm_ctx(tfm);
struct crypto_cipher *cipher;
cipher = crypto_spawn_cipher(spawn);
if (IS_ERR(cipher))
return PTR_ERR(cipher);
ctx->child = cipher;
return 0;
}
static void crypto_ctr_exit_tfm(struct crypto_tfm *tfm)
{
struct crypto_ctr_ctx *ctx = crypto_tfm_ctx(tfm);
crypto_free_cipher(ctx->child);
}
static struct crypto_instance *crypto_ctr_alloc(struct rtattr **tb)
{
struct crypto_instance *inst;
struct crypto_attr_type *algt;
struct crypto_alg *alg; struct crypto_alg *alg;
u32 mask;
int err; int err;
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_BLKCIPHER); inst = skcipher_alloc_instance_simple(tmpl, tb, &alg);
if (err) if (IS_ERR(inst))
return ERR_PTR(err); return PTR_ERR(inst);
algt = crypto_get_attr_type(tb);
if (IS_ERR(algt))
return ERR_CAST(algt);
mask = CRYPTO_ALG_TYPE_MASK |
crypto_requires_off(algt->type, algt->mask,
CRYPTO_ALG_NEED_FALLBACK);
alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER, mask);
if (IS_ERR(alg))
return ERR_CAST(alg);
/* Block size must be >= 4 bytes. */ /* Block size must be >= 4 bytes. */
err = -EINVAL; err = -EINVAL;
if (alg->cra_blocksize < 4) if (alg->cra_blocksize < 4)
goto out_put_alg; goto out_free_inst;
/* If this is false we'd fail the alignment of crypto_inc. */ /* If this is false we'd fail the alignment of crypto_inc. */
if (alg->cra_blocksize % 4) if (alg->cra_blocksize % 4)
goto out_put_alg; goto out_free_inst;
inst = crypto_alloc_instance("ctr", alg); /* CTR mode is a stream cipher. */
if (IS_ERR(inst)) inst->alg.base.cra_blocksize = 1;
goto out;
inst->alg.cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER; /*
inst->alg.cra_priority = alg->cra_priority; * To simplify the implementation, configure the skcipher walk to only
inst->alg.cra_blocksize = 1; * give a partial block at the very end, never earlier.
inst->alg.cra_alignmask = alg->cra_alignmask; */
inst->alg.cra_type = &crypto_blkcipher_type; inst->alg.chunksize = alg->cra_blocksize;
inst->alg.cra_blkcipher.ivsize = alg->cra_blocksize; inst->alg.encrypt = crypto_ctr_crypt;
inst->alg.cra_blkcipher.min_keysize = alg->cra_cipher.cia_min_keysize; inst->alg.decrypt = crypto_ctr_crypt;
inst->alg.cra_blkcipher.max_keysize = alg->cra_cipher.cia_max_keysize;
inst->alg.cra_ctxsize = sizeof(struct crypto_ctr_ctx); err = skcipher_register_instance(tmpl, inst);
if (err)
inst->alg.cra_init = crypto_ctr_init_tfm; goto out_free_inst;
inst->alg.cra_exit = crypto_ctr_exit_tfm; goto out_put_alg;
inst->alg.cra_blkcipher.setkey = crypto_ctr_setkey;
inst->alg.cra_blkcipher.encrypt = crypto_ctr_crypt;
inst->alg.cra_blkcipher.decrypt = crypto_ctr_crypt;
out:
crypto_mod_put(alg);
return inst;
out_free_inst:
inst->free(inst);
out_put_alg: out_put_alg:
inst = ERR_PTR(err); crypto_mod_put(alg);
goto out; return err;
} }
static void crypto_ctr_free(struct crypto_instance *inst)
{
crypto_drop_spawn(crypto_instance_ctx(inst));
kfree(inst);
}
static struct crypto_template crypto_ctr_tmpl = {
.name = "ctr",
.alloc = crypto_ctr_alloc,
.free = crypto_ctr_free,
.module = THIS_MODULE,
};
static int crypto_rfc3686_setkey(struct crypto_skcipher *parent, static int crypto_rfc3686_setkey(struct crypto_skcipher *parent,
const u8 *key, unsigned int keylen) const u8 *key, unsigned int keylen)
{ {
@ -444,42 +360,34 @@ err_free_inst:
goto out; goto out;
} }
static struct crypto_template crypto_rfc3686_tmpl = { static struct crypto_template crypto_ctr_tmpls[] = {
.name = "rfc3686", {
.create = crypto_rfc3686_create, .name = "ctr",
.module = THIS_MODULE, .create = crypto_ctr_create,
.module = THIS_MODULE,
}, {
.name = "rfc3686",
.create = crypto_rfc3686_create,
.module = THIS_MODULE,
},
}; };
static int __init crypto_ctr_module_init(void) static int __init crypto_ctr_module_init(void)
{ {
int err; return crypto_register_templates(crypto_ctr_tmpls,
ARRAY_SIZE(crypto_ctr_tmpls));
err = crypto_register_template(&crypto_ctr_tmpl);
if (err)
goto out;
err = crypto_register_template(&crypto_rfc3686_tmpl);
if (err)
goto out_drop_ctr;
out:
return err;
out_drop_ctr:
crypto_unregister_template(&crypto_ctr_tmpl);
goto out;
} }
static void __exit crypto_ctr_module_exit(void) static void __exit crypto_ctr_module_exit(void)
{ {
crypto_unregister_template(&crypto_rfc3686_tmpl); crypto_unregister_templates(crypto_ctr_tmpls,
crypto_unregister_template(&crypto_ctr_tmpl); ARRAY_SIZE(crypto_ctr_tmpls));
} }
module_init(crypto_ctr_module_init); module_init(crypto_ctr_module_init);
module_exit(crypto_ctr_module_exit); module_exit(crypto_ctr_module_exit);
MODULE_LICENSE("GPL"); MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("CTR Counter block mode"); MODULE_DESCRIPTION("CTR block cipher mode of operation");
MODULE_ALIAS_CRYPTO("rfc3686"); MODULE_ALIAS_CRYPTO("rfc3686");
MODULE_ALIAS_CRYPTO("ctr"); MODULE_ALIAS_CRYPTO("ctr");

4
crypto/des_generic.c
View File

@ -789,7 +789,7 @@ static int des_setkey(struct crypto_tfm *tfm, const u8 *key,
/* Expand to tmp */ /* Expand to tmp */
ret = des_ekey(tmp, key); ret = des_ekey(tmp, key);
if (unlikely(ret == 0) && (*flags & CRYPTO_TFM_REQ_WEAK_KEY)) { if (unlikely(ret == 0) && (*flags & CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) {
*flags |= CRYPTO_TFM_RES_WEAK_KEY; *flags |= CRYPTO_TFM_RES_WEAK_KEY;
return -EINVAL; return -EINVAL;
} }
@ -866,7 +866,7 @@ int __des3_ede_setkey(u32 *expkey, u32 *flags, const u8 *key,
if (unlikely(!((K[0] ^ K[2]) | (K[1] ^ K[3])) || if (unlikely(!((K[0] ^ K[2]) | (K[1] ^ K[3])) ||
!((K[2] ^ K[4]) | (K[3] ^ K[5]))) && !((K[2] ^ K[4]) | (K[3] ^ K[5]))) &&
(*flags & CRYPTO_TFM_REQ_WEAK_KEY)) { (*flags & CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) {
*flags |= CRYPTO_TFM_RES_WEAK_KEY; *flags |= CRYPTO_TFM_RES_WEAK_KEY;
return -EINVAL; return -EINVAL;
} }

151
crypto/ecb.c
View File

@ -11,162 +11,83 @@
*/ */
#include <crypto/algapi.h> #include <crypto/algapi.h>
#include <crypto/internal/skcipher.h>
#include <linux/err.h> #include <linux/err.h>
#include <linux/init.h> #include <linux/init.h>
#include <linux/kernel.h> #include <linux/kernel.h>
#include <linux/module.h> #include <linux/module.h>
#include <linux/scatterlist.h>
#include <linux/slab.h>
struct crypto_ecb_ctx { static int crypto_ecb_crypt(struct skcipher_request *req,
struct crypto_cipher *child; struct crypto_cipher *cipher,
};
static int crypto_ecb_setkey(struct crypto_tfm *parent, const u8 *key,
unsigned int keylen)
{
struct crypto_ecb_ctx *ctx = crypto_tfm_ctx(parent);
struct crypto_cipher *child = ctx->child;
int err;
crypto_cipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
crypto_cipher_set_flags(child, crypto_tfm_get_flags(parent) &
CRYPTO_TFM_REQ_MASK);
err = crypto_cipher_setkey(child, key, keylen);
crypto_tfm_set_flags(parent, crypto_cipher_get_flags(child) &
CRYPTO_TFM_RES_MASK);
return err;
}
static int crypto_ecb_crypt(struct blkcipher_desc *desc,
struct blkcipher_walk *walk,
struct crypto_cipher *tfm,
void (*fn)(struct crypto_tfm *, u8 *, const u8 *)) void (*fn)(struct crypto_tfm *, u8 *, const u8 *))
{ {
int bsize = crypto_cipher_blocksize(tfm); const unsigned int bsize = crypto_cipher_blocksize(cipher);
struct skcipher_walk walk;
unsigned int nbytes; unsigned int nbytes;
int err; int err;
err = blkcipher_walk_virt(desc, walk); err = skcipher_walk_virt(&walk, req, false);
while ((nbytes = walk->nbytes)) { while ((nbytes = walk.nbytes) != 0) {
u8 *wsrc = walk->src.virt.addr; const u8 *src = walk.src.virt.addr;
u8 *wdst = walk->dst.virt.addr; u8 *dst = walk.dst.virt.addr;
do { do {
fn(crypto_cipher_tfm(tfm), wdst, wsrc); fn(crypto_cipher_tfm(cipher), dst, src);
wsrc += bsize; src += bsize;
wdst += bsize; dst += bsize;
} while ((nbytes -= bsize) >= bsize); } while ((nbytes -= bsize) >= bsize);
err = blkcipher_walk_done(desc, walk, nbytes); err = skcipher_walk_done(&walk, nbytes);
} }
return err; return err;
} }
static int crypto_ecb_encrypt(struct blkcipher_desc *desc, static int crypto_ecb_encrypt(struct skcipher_request *req)
struct scatterlist *dst, struct scatterlist *src,
unsigned int nbytes)
{ {
struct blkcipher_walk walk; struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct crypto_blkcipher *tfm = desc->tfm; struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
struct crypto_ecb_ctx *ctx = crypto_blkcipher_ctx(tfm);
struct crypto_cipher *child = ctx->child;
blkcipher_walk_init(&walk, dst, src, nbytes); return crypto_ecb_crypt(req, cipher,
return crypto_ecb_crypt(desc, &walk, child, crypto_cipher_alg(cipher)->cia_encrypt);
crypto_cipher_alg(child)->cia_encrypt);
} }
static int crypto_ecb_decrypt(struct blkcipher_desc *desc, static int crypto_ecb_decrypt(struct skcipher_request *req)
struct scatterlist *dst, struct scatterlist *src,
unsigned int nbytes)
{ {
struct blkcipher_walk walk; struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct crypto_blkcipher *tfm = desc->tfm; struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
struct crypto_ecb_ctx *ctx = crypto_blkcipher_ctx(tfm);
struct crypto_cipher *child = ctx->child;
blkcipher_walk_init(&walk, dst, src, nbytes); return crypto_ecb_crypt(req, cipher,
return crypto_ecb_crypt(desc, &walk, child, crypto_cipher_alg(cipher)->cia_decrypt);
crypto_cipher_alg(child)->cia_decrypt);
} }
static int crypto_ecb_init_tfm(struct crypto_tfm *tfm) static int crypto_ecb_create(struct crypto_template *tmpl, struct rtattr **tb)
{ {
struct crypto_instance *inst = (void *)tfm->__crt_alg; struct skcipher_instance *inst;
struct crypto_spawn *spawn = crypto_instance_ctx(inst);
struct crypto_ecb_ctx *ctx = crypto_tfm_ctx(tfm);
struct crypto_cipher *cipher;
cipher = crypto_spawn_cipher(spawn);
if (IS_ERR(cipher))
return PTR_ERR(cipher);
ctx->child = cipher;
return 0;
}
static void crypto_ecb_exit_tfm(struct crypto_tfm *tfm)
{
struct crypto_ecb_ctx *ctx = crypto_tfm_ctx(tfm);
crypto_free_cipher(ctx->child);
}
static struct crypto_instance *crypto_ecb_alloc(struct rtattr **tb)
{
struct crypto_instance *inst;
struct crypto_alg *alg; struct crypto_alg *alg;
int err; int err;
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_BLKCIPHER); inst = skcipher_alloc_instance_simple(tmpl, tb, &alg);
if (err)
return ERR_PTR(err);
alg = crypto_get_attr_alg(tb, CRYPTO_ALG_TYPE_CIPHER,
CRYPTO_ALG_TYPE_MASK);
if (IS_ERR(alg))
return ERR_CAST(alg);
inst = crypto_alloc_instance("ecb", alg);
if (IS_ERR(inst)) if (IS_ERR(inst))
goto out_put_alg; return PTR_ERR(inst);
inst->alg.cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER; inst->alg.ivsize = 0; /* ECB mode doesn't take an IV */
inst->alg.cra_priority = alg->cra_priority;
inst->alg.cra_blocksize = alg->cra_blocksize;
inst->alg.cra_alignmask = alg->cra_alignmask;
inst->alg.cra_type = &crypto_blkcipher_type;
inst->alg.cra_blkcipher.min_keysize = alg->cra_cipher.cia_min_keysize; inst->alg.encrypt = crypto_ecb_encrypt;
inst->alg.cra_blkcipher.max_keysize = alg->cra_cipher.cia_max_keysize; inst->alg.decrypt = crypto_ecb_decrypt;
inst->alg.cra_ctxsize = sizeof(struct crypto_ecb_ctx); err = skcipher_register_instance(tmpl, inst);
if (err)
inst->alg.cra_init = crypto_ecb_init_tfm; inst->free(inst);
inst->alg.cra_exit = crypto_ecb_exit_tfm;
inst->alg.cra_blkcipher.setkey = crypto_ecb_setkey;
inst->alg.cra_blkcipher.encrypt = crypto_ecb_encrypt;
inst->alg.cra_blkcipher.decrypt = crypto_ecb_decrypt;
out_put_alg:
crypto_mod_put(alg); crypto_mod_put(alg);
return inst; return err;
}
static void crypto_ecb_free(struct crypto_instance *inst)
{
crypto_drop_spawn(crypto_instance_ctx(inst));
kfree(inst);
} }
static struct crypto_template crypto_ecb_tmpl = { static struct crypto_template crypto_ecb_tmpl = {
.name = "ecb", .name = "ecb",
.alloc = crypto_ecb_alloc, .create = crypto_ecb_create,
.free = crypto_ecb_free,
.module = THIS_MODULE, .module = THIS_MODULE,
}; };
@ -184,5 +105,5 @@ module_init(crypto_ecb_module_init);
module_exit(crypto_ecb_module_exit); module_exit(crypto_ecb_module_exit);
MODULE_LICENSE("GPL"); MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("ECB block cipher algorithm"); MODULE_DESCRIPTION("ECB block cipher mode of operation");
MODULE_ALIAS_CRYPTO("ecb"); MODULE_ALIAS_CRYPTO("ecb");

75
crypto/gcm.c
View File

@ -247,7 +247,7 @@ static int gcm_hash_len(struct aead_request *req, u32 flags)
struct crypto_gcm_req_priv_ctx *pctx = crypto_gcm_reqctx(req); struct crypto_gcm_req_priv_ctx *pctx = crypto_gcm_reqctx(req);
struct ahash_request *ahreq = &pctx->u.ahreq; struct ahash_request *ahreq = &pctx->u.ahreq;
struct crypto_gcm_ghash_ctx *gctx = &pctx->ghash_ctx; struct crypto_gcm_ghash_ctx *gctx = &pctx->ghash_ctx;
u128 lengths; be128 lengths;
lengths.a = cpu_to_be64(req->assoclen * 8); lengths.a = cpu_to_be64(req->assoclen * 8);
lengths.b = cpu_to_be64(gctx->cryptlen * 8); lengths.b = cpu_to_be64(gctx->cryptlen * 8);
@ -727,12 +727,6 @@ static int crypto_gcm_create(struct crypto_template *tmpl, struct rtattr **tb)
ctr_name, "ghash"); ctr_name, "ghash");
} }
static struct crypto_template crypto_gcm_tmpl = {
.name = "gcm",
.create = crypto_gcm_create,
.module = THIS_MODULE,
};
static int crypto_gcm_base_create(struct crypto_template *tmpl, static int crypto_gcm_base_create(struct crypto_template *tmpl,
struct rtattr **tb) struct rtattr **tb)
{ {
@ -756,12 +750,6 @@ static int crypto_gcm_base_create(struct crypto_template *tmpl,
ctr_name, ghash_name); ctr_name, ghash_name);
} }
static struct crypto_template crypto_gcm_base_tmpl = {
.name = "gcm_base",
.create = crypto_gcm_base_create,
.module = THIS_MODULE,
};
static int crypto_rfc4106_setkey(struct crypto_aead *parent, const u8 *key, static int crypto_rfc4106_setkey(struct crypto_aead *parent, const u8 *key,
unsigned int keylen) unsigned int keylen)
{ {
@ -989,12 +977,6 @@ out_free_inst:
goto out; goto out;
} }
static struct crypto_template crypto_rfc4106_tmpl = {
.name = "rfc4106",
.create = crypto_rfc4106_create,
.module = THIS_MODULE,
};
static int crypto_rfc4543_setkey(struct crypto_aead *parent, const u8 *key, static int crypto_rfc4543_setkey(struct crypto_aead *parent, const u8 *key,
unsigned int keylen) unsigned int keylen)
{ {
@ -1231,10 +1213,24 @@ out_free_inst:
goto out; goto out;
} }
static struct crypto_template crypto_rfc4543_tmpl = { static struct crypto_template crypto_gcm_tmpls[] = {
.name = "rfc4543", {
.create = crypto_rfc4543_create, .name = "gcm_base",
.module = THIS_MODULE, .create = crypto_gcm_base_create,
.module = THIS_MODULE,
}, {
.name = "gcm",
.create = crypto_gcm_create,
.module = THIS_MODULE,
}, {
.name = "rfc4106",
.create = crypto_rfc4106_create,
.module = THIS_MODULE,
}, {
.name = "rfc4543",
.create = crypto_rfc4543_create,
.module = THIS_MODULE,
},
}; };
static int __init crypto_gcm_module_init(void) static int __init crypto_gcm_module_init(void)
@ -1247,42 +1243,19 @@ static int __init crypto_gcm_module_init(void)
sg_init_one(&gcm_zeroes->sg, gcm_zeroes->buf, sizeof(gcm_zeroes->buf)); sg_init_one(&gcm_zeroes->sg, gcm_zeroes->buf, sizeof(gcm_zeroes->buf));
err = crypto_register_template(&crypto_gcm_base_tmpl); err = crypto_register_templates(crypto_gcm_tmpls,
ARRAY_SIZE(crypto_gcm_tmpls));
if (err) if (err)
goto out; kfree(gcm_zeroes);
err = crypto_register_template(&crypto_gcm_tmpl);
if (err)
goto out_undo_base;
err = crypto_register_template(&crypto_rfc4106_tmpl);
if (err)
goto out_undo_gcm;
err = crypto_register_template(&crypto_rfc4543_tmpl);
if (err)
goto out_undo_rfc4106;
return 0;
out_undo_rfc4106:
crypto_unregister_template(&crypto_rfc4106_tmpl);
out_undo_gcm:
crypto_unregister_template(&crypto_gcm_tmpl);
out_undo_base:
crypto_unregister_template(&crypto_gcm_base_tmpl);
out:
kfree(gcm_zeroes);
return err; return err;
} }
static void __exit crypto_gcm_module_exit(void) static void __exit crypto_gcm_module_exit(void)
{ {
kfree(gcm_zeroes); kfree(gcm_zeroes);
crypto_unregister_template(&crypto_rfc4543_tmpl); crypto_unregister_templates(crypto_gcm_tmpls,
crypto_unregister_template(&crypto_rfc4106_tmpl); ARRAY_SIZE(crypto_gcm_tmpls));
crypto_unregister_template(&crypto_gcm_tmpl);
crypto_unregister_template(&crypto_gcm_base_tmpl);
} }
module_init(crypto_gcm_module_init); module_init(crypto_gcm_module_init);

198
crypto/keywrap.c
View File

@ -56,7 +56,7 @@
* u8 *iv = data; * u8 *iv = data;
* u8 *pt = data + crypto_skcipher_ivsize(tfm); * u8 *pt = data + crypto_skcipher_ivsize(tfm);
* <ensure that pt contains the plaintext of size ptlen> * <ensure that pt contains the plaintext of size ptlen>
* sg_init_one(&sg, ptdata, ptlen); * sg_init_one(&sg, pt, ptlen);
* skcipher_request_set_crypt(req, &sg, &sg, ptlen, iv); * skcipher_request_set_crypt(req, &sg, &sg, ptlen, iv);
* *
* ==> After encryption, data now contains full KW result as per SP800-38F. * ==> After encryption, data now contains full KW result as per SP800-38F.
@ -70,8 +70,8 @@
* u8 *iv = data; * u8 *iv = data;
* u8 *ct = data + crypto_skcipher_ivsize(tfm); * u8 *ct = data + crypto_skcipher_ivsize(tfm);
* unsigned int ctlen = datalen - crypto_skcipher_ivsize(tfm); * unsigned int ctlen = datalen - crypto_skcipher_ivsize(tfm);
* sg_init_one(&sg, ctdata, ctlen); * sg_init_one(&sg, ct, ctlen);
* skcipher_request_set_crypt(req, &sg, &sg, ptlen, iv); * skcipher_request_set_crypt(req, &sg, &sg, ctlen, iv);
* *
* ==> After decryption (which hopefully does not return EBADMSG), the ct * ==> After decryption (which hopefully does not return EBADMSG), the ct
* pointer now points to the plaintext of size ctlen. * pointer now points to the plaintext of size ctlen.
@ -87,10 +87,6 @@
#include <crypto/scatterwalk.h> #include <crypto/scatterwalk.h>
#include <crypto/internal/skcipher.h> #include <crypto/internal/skcipher.h>
struct crypto_kw_ctx {
struct crypto_cipher *child;
};
struct crypto_kw_block { struct crypto_kw_block {
#define SEMIBSIZE 8 #define SEMIBSIZE 8
__be64 A; __be64 A;
@ -124,16 +120,13 @@ static void crypto_kw_scatterlist_ff(struct scatter_walk *walk,
} }
} }
static int crypto_kw_decrypt(struct blkcipher_desc *desc, static int crypto_kw_decrypt(struct skcipher_request *req)
struct scatterlist *dst, struct scatterlist *src,
unsigned int nbytes)
{ {
struct crypto_blkcipher *tfm = desc->tfm; struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct crypto_kw_ctx *ctx = crypto_blkcipher_ctx(tfm); struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
struct crypto_cipher *child = ctx->child;
struct crypto_kw_block block; struct crypto_kw_block block;
struct scatterlist *lsrc, *ldst; struct scatterlist *src, *dst;
u64 t = 6 * ((nbytes) >> 3); u64 t = 6 * ((req->cryptlen) >> 3);
unsigned int i; unsigned int i;
int ret = 0; int ret = 0;
@ -141,27 +134,27 @@ static int crypto_kw_decrypt(struct blkcipher_desc *desc,
* Require at least 2 semiblocks (note, the 3rd semiblock that is * Require at least 2 semiblocks (note, the 3rd semiblock that is
* required by SP800-38F is the IV. * required by SP800-38F is the IV.
*/ */
if (nbytes < (2 * SEMIBSIZE) || nbytes % SEMIBSIZE) if (req->cryptlen < (2 * SEMIBSIZE) || req->cryptlen % SEMIBSIZE)
return -EINVAL; return -EINVAL;
/* Place the IV into block A */ /* Place the IV into block A */
memcpy(&block.A, desc->info, SEMIBSIZE); memcpy(&block.A, req->iv, SEMIBSIZE);
/* /*
* src scatterlist is read-only. dst scatterlist is r/w. During the * src scatterlist is read-only. dst scatterlist is r/w. During the
* first loop, lsrc points to src and ldst to dst. For any * first loop, src points to req->src and dst to req->dst. For any
* subsequent round, the code operates on dst only. * subsequent round, the code operates on req->dst only.
*/ */
lsrc = src; src = req->src;
ldst = dst; dst = req->dst;
for (i = 0; i < 6; i++) { for (i = 0; i < 6; i++) {
struct scatter_walk src_walk, dst_walk; struct scatter_walk src_walk, dst_walk;
unsigned int tmp_nbytes = nbytes; unsigned int nbytes = req->cryptlen;
while (tmp_nbytes) { while (nbytes) {
/* move pointer by tmp_nbytes in the SGL */ /* move pointer by nbytes in the SGL */
crypto_kw_scatterlist_ff(&src_walk, lsrc, tmp_nbytes); crypto_kw_scatterlist_ff(&src_walk, src, nbytes);
/* get the source block */ /* get the source block */
scatterwalk_copychunks(&block.R, &src_walk, SEMIBSIZE, scatterwalk_copychunks(&block.R, &src_walk, SEMIBSIZE,
false); false);
@ -170,21 +163,21 @@ static int crypto_kw_decrypt(struct blkcipher_desc *desc,
block.A ^= cpu_to_be64(t); block.A ^= cpu_to_be64(t);
t--; t--;
/* perform KW operation: decrypt block */ /* perform KW operation: decrypt block */
crypto_cipher_decrypt_one(child, (u8*)&block, crypto_cipher_decrypt_one(cipher, (u8 *)&block,
(u8*)&block); (u8 *)&block);
/* move pointer by tmp_nbytes in the SGL */ /* move pointer by nbytes in the SGL */
crypto_kw_scatterlist_ff(&dst_walk, ldst, tmp_nbytes); crypto_kw_scatterlist_ff(&dst_walk, dst, nbytes);
/* Copy block->R into place */ /* Copy block->R into place */
scatterwalk_copychunks(&block.R, &dst_walk, SEMIBSIZE, scatterwalk_copychunks(&block.R, &dst_walk, SEMIBSIZE,
true); true);
tmp_nbytes -= SEMIBSIZE; nbytes -= SEMIBSIZE;
} }
/* we now start to operate on the dst SGL only */ /* we now start to operate on the dst SGL only */
lsrc = dst; src = req->dst;
ldst = dst; dst = req->dst;
} }
/* Perform authentication check */ /* Perform authentication check */
@ -196,15 +189,12 @@ static int crypto_kw_decrypt(struct blkcipher_desc *desc,
return ret; return ret;
} }
static int crypto_kw_encrypt(struct blkcipher_desc *desc, static int crypto_kw_encrypt(struct skcipher_request *req)
struct scatterlist *dst, struct scatterlist *src,
unsigned int nbytes)
{ {
struct crypto_blkcipher *tfm = desc->tfm; struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct crypto_kw_ctx *ctx = crypto_blkcipher_ctx(tfm); struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
struct crypto_cipher *child = ctx->child;
struct crypto_kw_block block; struct crypto_kw_block block;
struct scatterlist *lsrc, *ldst; struct scatterlist *src, *dst;
u64 t = 1; u64 t = 1;
unsigned int i; unsigned int i;
@ -214,7 +204,7 @@ static int crypto_kw_encrypt(struct blkcipher_desc *desc,
* This means that the dst memory must be one semiblock larger than src. * This means that the dst memory must be one semiblock larger than src.
* Also ensure that the given data is aligned to semiblock. * Also ensure that the given data is aligned to semiblock.
*/ */
if (nbytes < (2 * SEMIBSIZE) || nbytes % SEMIBSIZE) if (req->cryptlen < (2 * SEMIBSIZE) || req->cryptlen % SEMIBSIZE)
return -EINVAL; return -EINVAL;
/* /*
@ -225,26 +215,26 @@ static int crypto_kw_encrypt(struct blkcipher_desc *desc,
/* /*
* src scatterlist is read-only. dst scatterlist is r/w. During the * src scatterlist is read-only. dst scatterlist is r/w. During the
* first loop, lsrc points to src and ldst to dst. For any * first loop, src points to req->src and dst to req->dst. For any
* subsequent round, the code operates on dst only. * subsequent round, the code operates on req->dst only.
*/ */
lsrc = src; src = req->src;
ldst = dst; dst = req->dst;
for (i = 0; i < 6; i++) { for (i = 0; i < 6; i++) {
struct scatter_walk src_walk, dst_walk; struct scatter_walk src_walk, dst_walk;
unsigned int tmp_nbytes = nbytes; unsigned int nbytes = req->cryptlen;
scatterwalk_start(&src_walk, lsrc); scatterwalk_start(&src_walk, src);
scatterwalk_start(&dst_walk, ldst); scatterwalk_start(&dst_walk, dst);
while (tmp_nbytes) { while (nbytes) {
/* get the source block */ /* get the source block */
scatterwalk_copychunks(&block.R, &src_walk, SEMIBSIZE, scatterwalk_copychunks(&block.R, &src_walk, SEMIBSIZE,
false); false);
/* perform KW operation: encrypt block */ /* perform KW operation: encrypt block */
crypto_cipher_encrypt_one(child, (u8 *)&block, crypto_cipher_encrypt_one(cipher, (u8 *)&block,
(u8 *)&block); (u8 *)&block);
/* perform KW operation: modify IV with counter */ /* perform KW operation: modify IV with counter */
block.A ^= cpu_to_be64(t); block.A ^= cpu_to_be64(t);
@ -254,117 +244,59 @@ static int crypto_kw_encrypt(struct blkcipher_desc *desc,
scatterwalk_copychunks(&block.R, &dst_walk, SEMIBSIZE, scatterwalk_copychunks(&block.R, &dst_walk, SEMIBSIZE,
true); true);
tmp_nbytes -= SEMIBSIZE; nbytes -= SEMIBSIZE;
} }
/* we now start to operate on the dst SGL only */ /* we now start to operate on the dst SGL only */
lsrc = dst; src = req->dst;
ldst = dst; dst = req->dst;
} }
/* establish the IV for the caller to pick up */ /* establish the IV for the caller to pick up */
memcpy(desc->info, &block.A, SEMIBSIZE); memcpy(req->iv, &block.A, SEMIBSIZE);
memzero_explicit(&block, sizeof(struct crypto_kw_block)); memzero_explicit(&block, sizeof(struct crypto_kw_block));
return 0; return 0;
} }
static int crypto_kw_setkey(struct crypto_tfm *parent, const u8 *key, static int crypto_kw_create(struct crypto_template *tmpl, struct rtattr **tb)
unsigned int keylen)
{ {
struct crypto_kw_ctx *ctx = crypto_tfm_ctx(parent); struct skcipher_instance *inst;
struct crypto_cipher *child = ctx->child; struct crypto_alg *alg;
int err; int err;
crypto_cipher_clear_flags(child, CRYPTO_TFM_REQ_MASK); inst = skcipher_alloc_instance_simple(tmpl, tb, &alg);
crypto_cipher_set_flags(child, crypto_tfm_get_flags(parent) & if (IS_ERR(inst))
CRYPTO_TFM_REQ_MASK); return PTR_ERR(inst);
err = crypto_cipher_setkey(child, key, keylen);
crypto_tfm_set_flags(parent, crypto_cipher_get_flags(child) &
CRYPTO_TFM_RES_MASK);
return err;
}
static int crypto_kw_init_tfm(struct crypto_tfm *tfm) err = -EINVAL;
{
struct crypto_instance *inst = crypto_tfm_alg_instance(tfm);
struct crypto_spawn *spawn = crypto_instance_ctx(inst);
struct crypto_kw_ctx *ctx = crypto_tfm_ctx(tfm);
struct crypto_cipher *cipher;
cipher = crypto_spawn_cipher(spawn);
if (IS_ERR(cipher))
return PTR_ERR(cipher);
ctx->child = cipher;
return 0;
}
static void crypto_kw_exit_tfm(struct crypto_tfm *tfm)
{
struct crypto_kw_ctx *ctx = crypto_tfm_ctx(tfm);
crypto_free_cipher(ctx->child);
}
static struct crypto_instance *crypto_kw_alloc(struct rtattr **tb)
{
struct crypto_instance *inst = NULL;
struct crypto_alg *alg = NULL;
int err;
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_BLKCIPHER);
if (err)
return ERR_PTR(err);
alg = crypto_get_attr_alg(tb, CRYPTO_ALG_TYPE_CIPHER,
CRYPTO_ALG_TYPE_MASK);
if (IS_ERR(alg))
return ERR_CAST(alg);
inst = ERR_PTR(-EINVAL);
/* Section 5.1 requirement for KW */ /* Section 5.1 requirement for KW */
if (alg->cra_blocksize != sizeof(struct crypto_kw_block)) if (alg->cra_blocksize != sizeof(struct crypto_kw_block))
goto err; goto out_free_inst;
inst = crypto_alloc_instance("kw", alg); inst->alg.base.cra_blocksize = SEMIBSIZE;
if (IS_ERR(inst)) inst->alg.base.cra_alignmask = 0;
goto err; inst->alg.ivsize = SEMIBSIZE;
inst->alg.cra_flags = CRYPTO_ALG_TYPE_BLKCIPHER; inst->alg.encrypt = crypto_kw_encrypt;
inst->alg.cra_priority = alg->cra_priority; inst->alg.decrypt = crypto_kw_decrypt;
inst->alg.cra_blocksize = SEMIBSIZE;
inst->alg.cra_alignmask = 0;
inst->alg.cra_type = &crypto_blkcipher_type;
inst->alg.cra_blkcipher.ivsize = SEMIBSIZE;
inst->alg.cra_blkcipher.min_keysize = alg->cra_cipher.cia_min_keysize;
inst->alg.cra_blkcipher.max_keysize = alg->cra_cipher.cia_max_keysize;
inst->alg.cra_ctxsize = sizeof(struct crypto_kw_ctx); err = skcipher_register_instance(tmpl, inst);
if (err)
goto out_free_inst;
goto out_put_alg;
inst->alg.cra_init = crypto_kw_init_tfm; out_free_inst:
inst->alg.cra_exit = crypto_kw_exit_tfm; inst->free(inst);
out_put_alg:
inst->alg.cra_blkcipher.setkey = crypto_kw_setkey;
inst->alg.cra_blkcipher.encrypt = crypto_kw_encrypt;
inst->alg.cra_blkcipher.decrypt = crypto_kw_decrypt;
err:
crypto_mod_put(alg); crypto_mod_put(alg);
return inst; return err;
}
static void crypto_kw_free(struct crypto_instance *inst)
{
crypto_drop_spawn(crypto_instance_ctx(inst));
kfree(inst);
} }
static struct crypto_template crypto_kw_tmpl = { static struct crypto_template crypto_kw_tmpl = {
.name = "kw", .name = "kw",
.alloc = crypto_kw_alloc, .create = crypto_kw_create,
.free = crypto_kw_free,
.module = THIS_MODULE, .module = THIS_MODULE,
}; };

19
crypto/morus1280.c
View File

@ -1,13 +1,9 @@
// SPDX-License-Identifier: GPL-2.0-or-later
/* /*
* The MORUS-1280 Authenticated-Encryption Algorithm * The MORUS-1280 Authenticated-Encryption Algorithm
* *
* Copyright (c) 2016-2018 Ondrej Mosnacek <omosnacek@gmail.com> * Copyright (c) 2016-2018 Ondrej Mosnacek <omosnacek@gmail.com>
* Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
* Software Foundation; either version 2 of the License, or (at your option)
* any later version.
*/ */
#include <asm/unaligned.h> #include <asm/unaligned.h>
@ -366,18 +362,19 @@ static void crypto_morus1280_process_crypt(struct morus1280_state *state,
const struct morus1280_ops *ops) const struct morus1280_ops *ops)
{ {
struct skcipher_walk walk; struct skcipher_walk walk;
u8 *dst;
const u8 *src;
ops->skcipher_walk_init(&walk, req, false); ops->skcipher_walk_init(&walk, req, false);
while (walk.nbytes) { while (walk.nbytes) {
src = walk.src.virt.addr; unsigned int nbytes = walk.nbytes;
dst = walk.dst.virt.addr;
ops->crypt_chunk(state, dst, src, walk.nbytes); if (nbytes < walk.total)
nbytes = round_down(nbytes, walk.stride);
skcipher_walk_done(&walk, 0); ops->crypt_chunk(state, walk.dst.virt.addr, walk.src.virt.addr,
nbytes);
skcipher_walk_done(&walk, walk.nbytes - nbytes);
} }
} }

19
crypto/morus640.c
View File

@ -1,13 +1,9 @@
// SPDX-License-Identifier: GPL-2.0-or-later
/* /*
* The MORUS-640 Authenticated-Encryption Algorithm * The MORUS-640 Authenticated-Encryption Algorithm
* *
* Copyright (c) 2016-2018 Ondrej Mosnacek <omosnacek@gmail.com> * Copyright (c) 2016-2018 Ondrej Mosnacek <omosnacek@gmail.com>
* Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the Free
* Software Foundation; either version 2 of the License, or (at your option)
* any later version.
*/ */
#include <asm/unaligned.h> #include <asm/unaligned.h>
@ -365,18 +361,19 @@ static void crypto_morus640_process_crypt(struct morus640_state *state,
const struct morus640_ops *ops) const struct morus640_ops *ops)
{ {
struct skcipher_walk walk; struct skcipher_walk walk;
u8 *dst;
const u8 *src;
ops->skcipher_walk_init(&walk, req, false); ops->skcipher_walk_init(&walk, req, false);
while (walk.nbytes) { while (walk.nbytes) {
src = walk.src.virt.addr; unsigned int nbytes = walk.nbytes;
dst = walk.dst.virt.addr;
ops->crypt_chunk(state, dst, src, walk.nbytes); if (nbytes < walk.total)
nbytes = round_down(nbytes, walk.stride);
skcipher_walk_done(&walk, 0); ops->crypt_chunk(state, walk.dst.virt.addr, walk.src.virt.addr,
nbytes);
skcipher_walk_done(&walk, walk.nbytes - nbytes);
} }
} }

202
crypto/ofb.c
View File

@ -5,9 +5,6 @@
* *
* Copyright (C) 2018 ARM Limited or its affiliates. * Copyright (C) 2018 ARM Limited or its affiliates.
* All rights reserved. * All rights reserved.
*
* Based loosely on public domain code gleaned from libtomcrypt
* (https://github.com/libtom/libtomcrypt).
*/ */
#include <crypto/algapi.h> #include <crypto/algapi.h>
@ -16,189 +13,70 @@
#include <linux/init.h> #include <linux/init.h>
#include <linux/kernel.h> #include <linux/kernel.h>
#include <linux/module.h> #include <linux/module.h>
#include <linux/scatterlist.h>
#include <linux/slab.h>
struct crypto_ofb_ctx { static int crypto_ofb_crypt(struct skcipher_request *req)
struct crypto_cipher *child;
int cnt;
};
static int crypto_ofb_setkey(struct crypto_skcipher *parent, const u8 *key,
unsigned int keylen)
{ {
struct crypto_ofb_ctx *ctx = crypto_skcipher_ctx(parent); struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct crypto_cipher *child = ctx->child; struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
const unsigned int bsize = crypto_cipher_blocksize(cipher);
struct skcipher_walk walk;
int err; int err;
crypto_cipher_clear_flags(child, CRYPTO_TFM_REQ_MASK); err = skcipher_walk_virt(&walk, req, false);
crypto_cipher_set_flags(child, crypto_skcipher_get_flags(parent) &
CRYPTO_TFM_REQ_MASK);
err = crypto_cipher_setkey(child, key, keylen);
crypto_skcipher_set_flags(parent, crypto_cipher_get_flags(child) &
CRYPTO_TFM_RES_MASK);
return err;
}
static int crypto_ofb_encrypt_segment(struct crypto_ofb_ctx *ctx, while (walk.nbytes >= bsize) {
struct skcipher_walk *walk, const u8 *src = walk.src.virt.addr;
struct crypto_cipher *tfm) u8 *dst = walk.dst.virt.addr;
{ u8 * const iv = walk.iv;
int bsize = crypto_cipher_blocksize(tfm); unsigned int nbytes = walk.nbytes;
int nbytes = walk->nbytes;
u8 *src = walk->src.virt.addr; do {
u8 *dst = walk->dst.virt.addr; crypto_cipher_encrypt_one(cipher, iv, iv);
u8 *iv = walk->iv; crypto_xor_cpy(dst, src, iv, bsize);
dst += bsize;
src += bsize;
} while ((nbytes -= bsize) >= bsize);
do { err = skcipher_walk_done(&walk, nbytes);
if (ctx->cnt == bsize) {
if (nbytes < bsize)
break;
crypto_cipher_encrypt_one(tfm, iv, iv);
ctx->cnt = 0;
}
*dst = *src ^ iv[ctx->cnt];
src++;
dst++;
ctx->cnt++;
} while (--nbytes);
return nbytes;
}
static int crypto_ofb_encrypt(struct skcipher_request *req)
{
struct skcipher_walk walk;
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
unsigned int bsize;
struct crypto_ofb_ctx *ctx = crypto_skcipher_ctx(tfm);
struct crypto_cipher *child = ctx->child;
int ret = 0;
bsize = crypto_cipher_blocksize(child);
ctx->cnt = bsize;
ret = skcipher_walk_virt(&walk, req, false);
while (walk.nbytes) {
ret = crypto_ofb_encrypt_segment(ctx, &walk, child);
ret = skcipher_walk_done(&walk, ret);
} }
return ret; if (walk.nbytes) {
} crypto_cipher_encrypt_one(cipher, walk.iv, walk.iv);
crypto_xor_cpy(walk.dst.virt.addr, walk.src.virt.addr, walk.iv,
/* OFB encrypt and decrypt are identical */ walk.nbytes);
static int crypto_ofb_decrypt(struct skcipher_request *req) err = skcipher_walk_done(&walk, 0);
{ }
return crypto_ofb_encrypt(req); return err;
}
static int crypto_ofb_init_tfm(struct crypto_skcipher *tfm)
{
struct skcipher_instance *inst = skcipher_alg_instance(tfm);
struct crypto_spawn *spawn = skcipher_instance_ctx(inst);
struct crypto_ofb_ctx *ctx = crypto_skcipher_ctx(tfm);
struct crypto_cipher *cipher;
cipher = crypto_spawn_cipher(spawn);
if (IS_ERR(cipher))
return PTR_ERR(cipher);
ctx->child = cipher;
return 0;
}
static void crypto_ofb_exit_tfm(struct crypto_skcipher *tfm)
{
struct crypto_ofb_ctx *ctx = crypto_skcipher_ctx(tfm);
crypto_free_cipher(ctx->child);
}
static void crypto_ofb_free(struct skcipher_instance *inst)
{
crypto_drop_skcipher(skcipher_instance_ctx(inst));
kfree(inst);
} }
static int crypto_ofb_create(struct crypto_template *tmpl, struct rtattr **tb) static int crypto_ofb_create(struct crypto_template *tmpl, struct rtattr **tb)
{ {
struct skcipher_instance *inst; struct skcipher_instance *inst;
struct crypto_attr_type *algt;
struct crypto_spawn *spawn;
struct crypto_alg *alg; struct crypto_alg *alg;
u32 mask;
int err; int err;
err = crypto_check_attr_type(tb, CRYPTO_ALG_TYPE_SKCIPHER); inst = skcipher_alloc_instance_simple(tmpl, tb, &alg);
if (err) if (IS_ERR(inst))
return err; return PTR_ERR(inst);
inst = kzalloc(sizeof(*inst) + sizeof(*spawn), GFP_KERNEL); /* OFB mode is a stream cipher. */
if (!inst) inst->alg.base.cra_blocksize = 1;
return -ENOMEM;
algt = crypto_get_attr_type(tb); /*
err = PTR_ERR(algt); * To simplify the implementation, configure the skcipher walk to only
if (IS_ERR(algt)) * give a partial block at the very end, never earlier.
goto err_free_inst; */
inst->alg.chunksize = alg->cra_blocksize;
mask = CRYPTO_ALG_TYPE_MASK | inst->alg.encrypt = crypto_ofb_crypt;
crypto_requires_off(algt->type, algt->mask, inst->alg.decrypt = crypto_ofb_crypt;
CRYPTO_ALG_NEED_FALLBACK);
alg = crypto_get_attr_alg(tb, CRYPTO_ALG_TYPE_CIPHER, mask);
err = PTR_ERR(alg);
if (IS_ERR(alg))
goto err_free_inst;
spawn = skcipher_instance_ctx(inst);
err = crypto_init_spawn(spawn, alg, skcipher_crypto_instance(inst),
CRYPTO_ALG_TYPE_MASK);
crypto_mod_put(alg);
if (err)
goto err_free_inst;
err = crypto_inst_setname(skcipher_crypto_instance(inst), "ofb", alg);
if (err)
goto err_drop_spawn;
inst->alg.base.cra_priority = alg->cra_priority;
inst->alg.base.cra_blocksize = alg->cra_blocksize;
inst->alg.base.cra_alignmask = alg->cra_alignmask;
/* We access the data as u32s when xoring. */
inst->alg.base.cra_alignmask |= __alignof__(u32) - 1;
inst->alg.ivsize = alg->cra_blocksize;
inst->alg.min_keysize = alg->cra_cipher.cia_min_keysize;
inst->alg.max_keysize = alg->cra_cipher.cia_max_keysize;
inst->alg.base.cra_ctxsize = sizeof(struct crypto_ofb_ctx);
inst->alg.init = crypto_ofb_init_tfm;
inst->alg.exit = crypto_ofb_exit_tfm;
inst->alg.setkey = crypto_ofb_setkey;
inst->alg.encrypt = crypto_ofb_encrypt;
inst->alg.decrypt = crypto_ofb_decrypt;
inst->free = crypto_ofb_free;
err = skcipher_register_instance(tmpl, inst); err = skcipher_register_instance(tmpl, inst);
if (err) if (err)
goto err_drop_spawn; inst->free(inst);
out: crypto_mod_put(alg);
return err; return err;
err_drop_spawn:
crypto_drop_spawn(spawn);
err_free_inst:
kfree(inst);
goto out;
} }
static struct crypto_template crypto_ofb_tmpl = { static struct crypto_template crypto_ofb_tmpl = {
@ -221,5 +99,5 @@ module_init(crypto_ofb_module_init);
module_exit(crypto_ofb_module_exit); module_exit(crypto_ofb_module_exit);
MODULE_LICENSE("GPL"); MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("OFB block cipher algorithm"); MODULE_DESCRIPTION("OFB block cipher mode of operation");
MODULE_ALIAS_CRYPTO("ofb"); MODULE_ALIAS_CRYPTO("ofb");

143
crypto/pcbc.c
View File

@ -20,28 +20,6 @@
#include <linux/init.h> #include <linux/init.h>
#include <linux/kernel.h> #include <linux/kernel.h>
#include <linux/module.h> #include <linux/module.h>
#include <linux/slab.h>
#include <linux/compiler.h>
struct crypto_pcbc_ctx {
struct crypto_cipher *child;
};
static int crypto_pcbc_setkey(struct crypto_skcipher *parent, const u8 *key,
unsigned int keylen)
{
struct crypto_pcbc_ctx *ctx = crypto_skcipher_ctx(parent);
struct crypto_cipher *child = ctx->child;
int err;
crypto_cipher_clear_flags(child, CRYPTO_TFM_REQ_MASK);
crypto_cipher_set_flags(child, crypto_skcipher_get_flags(parent) &
CRYPTO_TFM_REQ_MASK);
err = crypto_cipher_setkey(child, key, keylen);
crypto_skcipher_set_flags(parent, crypto_cipher_get_flags(child) &
CRYPTO_TFM_RES_MASK);
return err;
}
static int crypto_pcbc_encrypt_segment(struct skcipher_request *req, static int crypto_pcbc_encrypt_segment(struct skcipher_request *req,
struct skcipher_walk *walk, struct skcipher_walk *walk,
@ -51,7 +29,7 @@ static int crypto_pcbc_encrypt_segment(struct skcipher_request *req,
unsigned int nbytes = walk->nbytes; unsigned int nbytes = walk->nbytes;
u8 *src = walk->src.virt.addr; u8 *src = walk->src.virt.addr;
u8 *dst = walk->dst.virt.addr; u8 *dst = walk->dst.virt.addr;
u8 *iv = walk->iv; u8 * const iv = walk->iv;
do { do {
crypto_xor(iv, src, bsize); crypto_xor(iv, src, bsize);
@ -72,7 +50,7 @@ static int crypto_pcbc_encrypt_inplace(struct skcipher_request *req,
int bsize = crypto_cipher_blocksize(tfm); int bsize = crypto_cipher_blocksize(tfm);
unsigned int nbytes = walk->nbytes; unsigned int nbytes = walk->nbytes;
u8 *src = walk->src.virt.addr; u8 *src = walk->src.virt.addr;
u8 *iv = walk->iv; u8 * const iv = walk->iv;
u8 tmpbuf[MAX_CIPHER_BLOCKSIZE]; u8 tmpbuf[MAX_CIPHER_BLOCKSIZE];
do { do {
@ -84,16 +62,13 @@ static int crypto_pcbc_encrypt_inplace(struct skcipher_request *req,
src += bsize; src += bsize;
} while ((nbytes -= bsize) >= bsize); } while ((nbytes -= bsize) >= bsize);
memcpy(walk->iv, iv, bsize);
return nbytes; return nbytes;
} }
static int crypto_pcbc_encrypt(struct skcipher_request *req) static int crypto_pcbc_encrypt(struct skcipher_request *req)
{ {
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct crypto_pcbc_ctx *ctx = crypto_skcipher_ctx(tfm); struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
struct crypto_cipher *child = ctx->child;
struct skcipher_walk walk; struct skcipher_walk walk;
unsigned int nbytes; unsigned int nbytes;
int err; int err;
@ -103,10 +78,10 @@ static int crypto_pcbc_encrypt(struct skcipher_request *req)
while ((nbytes = walk.nbytes)) { while ((nbytes = walk.nbytes)) {
if (walk.src.virt.addr == walk.dst.virt.addr) if (walk.src.virt.addr == walk.dst.virt.addr)
nbytes = crypto_pcbc_encrypt_inplace(req, &walk, nbytes = crypto_pcbc_encrypt_inplace(req, &walk,
child); cipher);
else else
nbytes = crypto_pcbc_encrypt_segment(req, &walk, nbytes = crypto_pcbc_encrypt_segment(req, &walk,
child); cipher);
err = skcipher_walk_done(&walk, nbytes); err = skcipher_walk_done(&walk, nbytes);
} }
@ -121,7 +96,7 @@ static int crypto_pcbc_decrypt_segment(struct skcipher_request *req,
unsigned int nbytes = walk->nbytes; unsigned int nbytes = walk->nbytes;
u8 *src = walk->src.virt.addr; u8 *src = walk->src.virt.addr;
u8 *dst = walk->dst.virt.addr; u8 *dst = walk->dst.virt.addr;
u8 *iv = walk->iv; u8 * const iv = walk->iv;
do { do {
crypto_cipher_decrypt_one(tfm, dst, src); crypto_cipher_decrypt_one(tfm, dst, src);
@ -132,8 +107,6 @@ static int crypto_pcbc_decrypt_segment(struct skcipher_request *req,
dst += bsize; dst += bsize;
} while ((nbytes -= bsize) >= bsize); } while ((nbytes -= bsize) >= bsize);
memcpy(walk->iv, iv, bsize);
return nbytes; return nbytes;
} }
@ -144,7 +117,7 @@ static int crypto_pcbc_decrypt_inplace(struct skcipher_request *req,
int bsize = crypto_cipher_blocksize(tfm); int bsize = crypto_cipher_blocksize(tfm);
unsigned int nbytes = walk->nbytes; unsigned int nbytes = walk->nbytes;
u8 *src = walk->src.virt.addr; u8 *src = walk->src.virt.addr;
u8 *iv = walk->iv; u8 * const iv = walk->iv;
u8 tmpbuf[MAX_CIPHER_BLOCKSIZE] __aligned(__alignof__(u32)); u8 tmpbuf[MAX_CIPHER_BLOCKSIZE] __aligned(__alignof__(u32));
do { do {
@ -156,16 +129,13 @@ static int crypto_pcbc_decrypt_inplace(struct skcipher_request *req,
src += bsize; src += bsize;
} while ((nbytes -= bsize) >= bsize); } while ((nbytes -= bsize) >= bsize);
memcpy(walk->iv, iv, bsize);
return nbytes; return nbytes;
} }
static int crypto_pcbc_decrypt(struct skcipher_request *req) static int crypto_pcbc_decrypt(struct skcipher_request *req)
{ {
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req); struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
struct crypto_pcbc_ctx *ctx = crypto_skcipher_ctx(tfm); struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
struct crypto_cipher *child = ctx->child;
struct skcipher_walk walk; struct skcipher_walk walk;
unsigned int nbytes; unsigned int nbytes;
int err; int err;
@ -175,117 +145,34 @@ static int crypto_pcbc_decrypt(struct skcipher_request *req)
while ((nbytes = walk.nbytes)) { while ((nbytes = walk.nbytes)) {
if (walk.src.virt.addr == walk.dst.virt.addr) if (walk.src.virt.addr == walk.dst.virt.addr)
nbytes = crypto_pcbc_decrypt_inplace(req, &walk, nbytes = crypto_pcbc_decrypt_inplace(req, &walk,
child); cipher);
else else
nbytes = crypto_pcbc_decrypt_segment(req, &walk, nbytes = crypto_pcbc_decrypt_segment(req, &walk,
child); cipher);
err = skcipher_walk_done(&walk, nbytes); err = skcipher_walk_done(&walk, nbytes);
} }
return err; return err;
} }
static int crypto_pcbc_init_tfm(struct crypto_skcipher *tfm)
{
struct skcipher_instance *inst = skcipher_alg_instance(tfm);
struct crypto_spawn *spawn = skcipher_instance_ctx(inst);
struct crypto_pcbc_ctx *ctx = crypto_skcipher_ctx(tfm);
struct crypto_cipher *cipher;
cipher = crypto_spawn_cipher(spawn);
if (IS_ERR(cipher))
return PTR_ERR(cipher);
ctx->child = cipher;
return 0;
}
static void crypto_pcbc_exit_tfm(struct crypto_skcipher *tfm)
{
struct crypto_pcbc_ctx *ctx = crypto_skcipher_ctx(tfm);
crypto_free_cipher(ctx->child);
}
static void crypto_pcbc_free(struct skcipher_instance *inst)
{
crypto_drop_skcipher(skcipher_instance_ctx(inst));
kfree(inst);
}
static int crypto_pcbc_create(struct crypto_template *tmpl, struct rtattr **tb) static int crypto_pcbc_create(struct crypto_template *tmpl, struct rtattr **tb)
{ {
struct skcipher_instance *inst; struct skcipher_instance *inst;
struct crypto_attr_type *algt;
struct crypto_spawn *spawn;
struct crypto_alg *alg; struct crypto_alg *alg;
int err; int err;
algt = crypto_get_attr_type(tb); inst = skcipher_alloc_instance_simple(tmpl, tb, &alg);
if (IS_ERR(algt)) if (IS_ERR(inst))
return PTR_ERR(algt); return PTR_ERR(inst);
if (((algt->type ^ CRYPTO_ALG_TYPE_SKCIPHER) & algt->mask) &
~CRYPTO_ALG_INTERNAL)
return -EINVAL;
inst = kzalloc(sizeof(*inst) + sizeof(*spawn), GFP_KERNEL);
if (!inst)
return -ENOMEM;
alg = crypto_get_attr_alg(tb, CRYPTO_ALG_TYPE_CIPHER |
(algt->type & CRYPTO_ALG_INTERNAL),
CRYPTO_ALG_TYPE_MASK |
(algt->mask & CRYPTO_ALG_INTERNAL));
err = PTR_ERR(alg);
if (IS_ERR(alg))
goto err_free_inst;
spawn = skcipher_instance_ctx(inst);
err = crypto_init_spawn(spawn, alg, skcipher_crypto_instance(inst),
CRYPTO_ALG_TYPE_MASK);
if (err)
goto err_put_alg;
err = crypto_inst_setname(skcipher_crypto_instance(inst), "pcbc", alg);
if (err)
goto err_drop_spawn;
inst->alg.base.cra_flags = alg->cra_flags & CRYPTO_ALG_INTERNAL;
inst->alg.base.cra_priority = alg->cra_priority;
inst->alg.base.cra_blocksize = alg->cra_blocksize;
inst->alg.base.cra_alignmask = alg->cra_alignmask;
inst->alg.ivsize = alg->cra_blocksize;
inst->alg.min_keysize = alg->cra_cipher.cia_min_keysize;
inst->alg.max_keysize = alg->cra_cipher.cia_max_keysize;
inst->alg.base.cra_ctxsize = sizeof(struct crypto_pcbc_ctx);
inst->alg.init = crypto_pcbc_init_tfm;
inst->alg.exit = crypto_pcbc_exit_tfm;
inst->alg.setkey = crypto_pcbc_setkey;
inst->alg.encrypt = crypto_pcbc_encrypt; inst->alg.encrypt = crypto_pcbc_encrypt;
inst->alg.decrypt = crypto_pcbc_decrypt; inst->alg.decrypt = crypto_pcbc_decrypt;
inst->free = crypto_pcbc_free;
err = skcipher_register_instance(tmpl, inst); err = skcipher_register_instance(tmpl, inst);
if (err) if (err)
goto err_drop_spawn; inst->free(inst);
crypto_mod_put(alg); crypto_mod_put(alg);
out:
return err; return err;
err_drop_spawn:
crypto_drop_spawn(spawn);
err_put_alg:
crypto_mod_put(alg);
err_free_inst:
kfree(inst);
goto out;
} }
static struct crypto_template crypto_pcbc_tmpl = { static struct crypto_template crypto_pcbc_tmpl = {
@ -308,5 +195,5 @@ module_init(crypto_pcbc_module_init);
module_exit(crypto_pcbc_module_exit); module_exit(crypto_pcbc_module_exit);
MODULE_LICENSE("GPL"); MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("PCBC block cipher algorithm"); MODULE_DESCRIPTION("PCBC block cipher mode of operation");
MODULE_ALIAS_CRYPTO("pcbc"); MODULE_ALIAS_CRYPTO("pcbc");

1
crypto/rsa-pkcs1pad.c
View File

@ -12,6 +12,7 @@
#include <crypto/algapi.h> #include <crypto/algapi.h>
#include <crypto/akcipher.h> #include <crypto/akcipher.h>
#include <crypto/internal/akcipher.h> #include <crypto/internal/akcipher.h>
#include <crypto/internal/rsa.h>
#include <linux/err.h> #include <linux/err.h>
#include <linux/init.h> #include <linux/init.h>
#include <linux/kernel.h> #include <linux/kernel.h>

7
crypto/seqiv.c
View File

@ -89,13 +89,12 @@ static int seqiv_aead_encrypt(struct aead_request *req)
if (unlikely(!IS_ALIGNED((unsigned long)info, if (unlikely(!IS_ALIGNED((unsigned long)info,
crypto_aead_alignmask(geniv) + 1))) { crypto_aead_alignmask(geniv) + 1))) {
info = kmalloc(ivsize, req->base.flags & info = kmemdup(req->iv, ivsize, req->base.flags &
CRYPTO_TFM_REQ_MAY_SLEEP ? GFP_KERNEL: CRYPTO_TFM_REQ_MAY_SLEEP ? GFP_KERNEL :
GFP_ATOMIC); GFP_ATOMIC);
if (!info) if (!info)
return -ENOMEM; return -ENOMEM;
memcpy(info, req->iv, ivsize);
compl = seqiv_aead_encrypt_complete; compl = seqiv_aead_encrypt_complete;
data = req; data = req;
} }

27
crypto/shash.c
View File

@ -53,6 +53,13 @@ static int shash_setkey_unaligned(struct crypto_shash *tfm, const u8 *key,
return err; return err;
} }
static void shash_set_needkey(struct crypto_shash *tfm, struct shash_alg *alg)
{
if (crypto_shash_alg_has_setkey(alg) &&
!(alg->base.cra_flags & CRYPTO_ALG_OPTIONAL_KEY))
crypto_shash_set_flags(tfm, CRYPTO_TFM_NEED_KEY);
}
int crypto_shash_setkey(struct crypto_shash *tfm, const u8 *key, int crypto_shash_setkey(struct crypto_shash *tfm, const u8 *key,
unsigned int keylen) unsigned int keylen)
{ {
@ -65,8 +72,10 @@ int crypto_shash_setkey(struct crypto_shash *tfm, const u8 *key,
else else
err = shash->setkey(tfm, key, keylen); err = shash->setkey(tfm, key, keylen);
if (err) if (unlikely(err)) {
shash_set_needkey(tfm, shash);
return err; return err;
}
crypto_shash_clear_flags(tfm, CRYPTO_TFM_NEED_KEY); crypto_shash_clear_flags(tfm, CRYPTO_TFM_NEED_KEY);
return 0; return 0;
@ -373,15 +382,14 @@ int crypto_init_shash_ops_async(struct crypto_tfm *tfm)
crt->final = shash_async_final; crt->final = shash_async_final;
crt->finup = shash_async_finup; crt->finup = shash_async_finup;
crt->digest = shash_async_digest; crt->digest = shash_async_digest;
crt->setkey = shash_async_setkey; if (crypto_shash_alg_has_setkey(alg))
crt->setkey = shash_async_setkey;
crypto_ahash_set_flags(crt, crypto_shash_get_flags(shash) & crypto_ahash_set_flags(crt, crypto_shash_get_flags(shash) &
CRYPTO_TFM_NEED_KEY); CRYPTO_TFM_NEED_KEY);
if (alg->export) crt->export = shash_async_export;
crt->export = shash_async_export; crt->import = shash_async_import;
if (alg->import)
crt->import = shash_async_import;
crt->reqsize = sizeof(struct shash_desc) + crypto_shash_descsize(shash); crt->reqsize = sizeof(struct shash_desc) + crypto_shash_descsize(shash);
@ -395,9 +403,7 @@ static int crypto_shash_init_tfm(struct crypto_tfm *tfm)
hash->descsize = alg->descsize; hash->descsize = alg->descsize;
if (crypto_shash_alg_has_setkey(alg) && shash_set_needkey(hash, alg);
!(alg->base.cra_flags & CRYPTO_ALG_OPTIONAL_KEY))
crypto_shash_set_flags(hash, CRYPTO_TFM_NEED_KEY);
return 0; return 0;
} }
@ -464,6 +470,9 @@ static int shash_prepare_alg(struct shash_alg *alg)
alg->statesize > HASH_MAX_STATESIZE) alg->statesize > HASH_MAX_STATESIZE)
return -EINVAL; return -EINVAL;
if ((alg->export && !alg->import) || (alg->import && !alg->export))
return -EINVAL;
base->cra_type = &crypto_shash_type; base->cra_type = &crypto_shash_type;
base->cra_flags &= ~CRYPTO_ALG_TYPE_MASK; base->cra_flags &= ~CRYPTO_ALG_TYPE_MASK;
base->cra_flags |= CRYPTO_ALG_TYPE_SHASH; base->cra_flags |= CRYPTO_ALG_TYPE_SHASH;

158
crypto/skcipher.c
View File

@ -585,6 +585,12 @@ static unsigned int crypto_skcipher_extsize(struct crypto_alg *alg)
return crypto_alg_extsize(alg); return crypto_alg_extsize(alg);
} }
static void skcipher_set_needkey(struct crypto_skcipher *tfm)
{
if (tfm->keysize)
crypto_skcipher_set_flags(tfm, CRYPTO_TFM_NEED_KEY);
}
static int skcipher_setkey_blkcipher(struct crypto_skcipher *tfm, static int skcipher_setkey_blkcipher(struct crypto_skcipher *tfm,
const u8 *key, unsigned int keylen) const u8 *key, unsigned int keylen)
{ {
@ -598,8 +604,10 @@ static int skcipher_setkey_blkcipher(struct crypto_skcipher *tfm,
err = crypto_blkcipher_setkey(blkcipher, key, keylen); err = crypto_blkcipher_setkey(blkcipher, key, keylen);
crypto_skcipher_set_flags(tfm, crypto_blkcipher_get_flags(blkcipher) & crypto_skcipher_set_flags(tfm, crypto_blkcipher_get_flags(blkcipher) &
CRYPTO_TFM_RES_MASK); CRYPTO_TFM_RES_MASK);
if (err) if (unlikely(err)) {
skcipher_set_needkey(tfm);
return err; return err;
}
crypto_skcipher_clear_flags(tfm, CRYPTO_TFM_NEED_KEY); crypto_skcipher_clear_flags(tfm, CRYPTO_TFM_NEED_KEY);
return 0; return 0;
@ -677,8 +685,7 @@ static int crypto_init_skcipher_ops_blkcipher(struct crypto_tfm *tfm)
skcipher->ivsize = crypto_blkcipher_ivsize(blkcipher); skcipher->ivsize = crypto_blkcipher_ivsize(blkcipher);
skcipher->keysize = calg->cra_blkcipher.max_keysize; skcipher->keysize = calg->cra_blkcipher.max_keysize;
if (skcipher->keysize) skcipher_set_needkey(skcipher);
crypto_skcipher_set_flags(skcipher, CRYPTO_TFM_NEED_KEY);
return 0; return 0;
} }
@ -698,8 +705,10 @@ static int skcipher_setkey_ablkcipher(struct crypto_skcipher *tfm,
crypto_skcipher_set_flags(tfm, crypto_skcipher_set_flags(tfm,
crypto_ablkcipher_get_flags(ablkcipher) & crypto_ablkcipher_get_flags(ablkcipher) &
CRYPTO_TFM_RES_MASK); CRYPTO_TFM_RES_MASK);
if (err) if (unlikely(err)) {
skcipher_set_needkey(tfm);
return err; return err;
}
crypto_skcipher_clear_flags(tfm, CRYPTO_TFM_NEED_KEY); crypto_skcipher_clear_flags(tfm, CRYPTO_TFM_NEED_KEY);
return 0; return 0;
@ -776,8 +785,7 @@ static int crypto_init_skcipher_ops_ablkcipher(struct crypto_tfm *tfm)
sizeof(struct ablkcipher_request); sizeof(struct ablkcipher_request);
skcipher->keysize = calg->cra_ablkcipher.max_keysize; skcipher->keysize = calg->cra_ablkcipher.max_keysize;
if (skcipher->keysize) skcipher_set_needkey(skcipher);
crypto_skcipher_set_flags(skcipher, CRYPTO_TFM_NEED_KEY);
return 0; return 0;
} }
@ -820,8 +828,10 @@ static int skcipher_setkey(struct crypto_skcipher *tfm, const u8 *key,
else else
err = cipher->setkey(tfm, key, keylen); err = cipher->setkey(tfm, key, keylen);
if (err) if (unlikely(err)) {
skcipher_set_needkey(tfm);
return err; return err;
}
crypto_skcipher_clear_flags(tfm, CRYPTO_TFM_NEED_KEY); crypto_skcipher_clear_flags(tfm, CRYPTO_TFM_NEED_KEY);
return 0; return 0;
@ -852,8 +862,7 @@ static int crypto_skcipher_init_tfm(struct crypto_tfm *tfm)
skcipher->ivsize = alg->ivsize; skcipher->ivsize = alg->ivsize;
skcipher->keysize = alg->max_keysize; skcipher->keysize = alg->max_keysize;
if (skcipher->keysize) skcipher_set_needkey(skcipher);
crypto_skcipher_set_flags(skcipher, CRYPTO_TFM_NEED_KEY);
if (alg->exit) if (alg->exit)
skcipher->base.exit = crypto_skcipher_exit_tfm; skcipher->base.exit = crypto_skcipher_exit_tfm;
@ -1058,5 +1067,136 @@ int skcipher_register_instance(struct crypto_template *tmpl,
} }
EXPORT_SYMBOL_GPL(skcipher_register_instance); EXPORT_SYMBOL_GPL(skcipher_register_instance);
static int skcipher_setkey_simple(struct crypto_skcipher *tfm, const u8 *key,
unsigned int keylen)
{
struct crypto_cipher *cipher = skcipher_cipher_simple(tfm);
int err;
crypto_cipher_clear_flags(cipher, CRYPTO_TFM_REQ_MASK);
crypto_cipher_set_flags(cipher, crypto_skcipher_get_flags(tfm) &
CRYPTO_TFM_REQ_MASK);
err = crypto_cipher_setkey(cipher, key, keylen);
crypto_skcipher_set_flags(tfm, crypto_cipher_get_flags(cipher) &
CRYPTO_TFM_RES_MASK);
return err;
}
static int skcipher_init_tfm_simple(struct crypto_skcipher *tfm)
{
struct skcipher_instance *inst = skcipher_alg_instance(tfm);
struct crypto_spawn *spawn = skcipher_instance_ctx(inst);
struct skcipher_ctx_simple *ctx = crypto_skcipher_ctx(tfm);
struct crypto_cipher *cipher;
cipher = crypto_spawn_cipher(spawn);
if (IS_ERR(cipher))
return PTR_ERR(cipher);
ctx->cipher = cipher;
return 0;
}
static void skcipher_exit_tfm_simple(struct crypto_skcipher *tfm)
{
struct skcipher_ctx_simple *ctx = crypto_skcipher_ctx(tfm);
crypto_free_cipher(ctx->cipher);
}
static void skcipher_free_instance_simple(struct skcipher_instance *inst)
{
crypto_drop_spawn(skcipher_instance_ctx(inst));
kfree(inst);
}
/**
* skcipher_alloc_instance_simple - allocate instance of simple block cipher mode
*
* Allocate an skcipher_instance for a simple block cipher mode of operation,
* e.g. cbc or ecb. The instance context will have just a single crypto_spawn,
* that for the underlying cipher. The {min,max}_keysize, ivsize, blocksize,
* alignmask, and priority are set from the underlying cipher but can be
* overridden if needed. The tfm context defaults to skcipher_ctx_simple, and
* default ->setkey(), ->init(), and ->exit() methods are installed.
*
* @tmpl: the template being instantiated
* @tb: the template parameters
* @cipher_alg_ret: on success, a pointer to the underlying cipher algorithm is
* returned here. It must be dropped with crypto_mod_put().
*
* Return: a pointer to the new instance, or an ERR_PTR(). The caller still
* needs to register the instance.
*/
struct skcipher_instance *
skcipher_alloc_instance_simple(struct crypto_template *tmpl, struct rtattr **tb,
struct crypto_alg **cipher_alg_ret)
{
struct crypto_attr_type *algt;
struct crypto_alg *cipher_alg;
struct skcipher_instance *inst;
struct crypto_spawn *spawn;
u32 mask;
int err;
algt = crypto_get_attr_type(tb);
if (IS_ERR(algt))
return ERR_CAST(algt);
if ((algt->type ^ CRYPTO_ALG_TYPE_SKCIPHER) & algt->mask)
return ERR_PTR(-EINVAL);
mask = CRYPTO_ALG_TYPE_MASK |
crypto_requires_off(algt->type, algt->mask,
CRYPTO_ALG_NEED_FALLBACK);
cipher_alg = crypto_get_attr_alg(tb, CRYPTO_ALG_TYPE_CIPHER, mask);
if (IS_ERR(cipher_alg))
return ERR_CAST(cipher_alg);
inst = kzalloc(sizeof(*inst) + sizeof(*spawn), GFP_KERNEL);
if (!inst) {
err = -ENOMEM;
goto err_put_cipher_alg;
}
spawn = skcipher_instance_ctx(inst);
err = crypto_inst_setname(skcipher_crypto_instance(inst), tmpl->name,
cipher_alg);
if (err)
goto err_free_inst;
err = crypto_init_spawn(spawn, cipher_alg,
skcipher_crypto_instance(inst),
CRYPTO_ALG_TYPE_MASK);
if (err)
goto err_free_inst;
inst->free = skcipher_free_instance_simple;
/* Default algorithm properties, can be overridden */
inst->alg.base.cra_blocksize = cipher_alg->cra_blocksize;
inst->alg.base.cra_alignmask = cipher_alg->cra_alignmask;
inst->alg.base.cra_priority = cipher_alg->cra_priority;
inst->alg.min_keysize = cipher_alg->cra_cipher.cia_min_keysize;
inst->alg.max_keysize = cipher_alg->cra_cipher.cia_max_keysize;
inst->alg.ivsize = cipher_alg->cra_blocksize;
/* Use skcipher_ctx_simple by default, can be overridden */
inst->alg.base.cra_ctxsize = sizeof(struct skcipher_ctx_simple);
inst->alg.setkey = skcipher_setkey_simple;
inst->alg.init = skcipher_init_tfm_simple;
inst->alg.exit = skcipher_exit_tfm_simple;
*cipher_alg_ret = cipher_alg;
return inst;
err_free_inst:
kfree(inst);
err_put_cipher_alg:
crypto_mod_put(cipher_alg);
return ERR_PTR(err);
}
EXPORT_SYMBOL_GPL(skcipher_alloc_instance_simple);
MODULE_LICENSE("GPL"); MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Symmetric key cipher type"); MODULE_DESCRIPTION("Symmetric key cipher type");

2
crypto/streebog_generic.c
View File

@ -960,7 +960,7 @@ static int streebog_init(struct shash_desc *desc)
memset(ctx, 0, sizeof(struct streebog_state)); memset(ctx, 0, sizeof(struct streebog_state));
for (i = 0; i < 8; i++) { for (i = 0; i < 8; i++) {
if (digest_size == STREEBOG256_DIGEST_SIZE) if (digest_size == STREEBOG256_DIGEST_SIZE)
ctx->h.qword[i] = 0x0101010101010101ULL; ctx->h.qword[i] = cpu_to_le64(0x0101010101010101ULL);
} }
return 0; return 0;
} }

2932
crypto/testmgr.c
View File

File diff suppressed because it is too large Load Diff

10841
crypto/testmgr.h
View File

File diff suppressed because it is too large Load Diff

6
crypto/tgr192.c
View File

@ -25,8 +25,9 @@
#include <linux/init.h> #include <linux/init.h>
#include <linux/module.h> #include <linux/module.h>
#include <linux/mm.h> #include <linux/mm.h>
#include <asm/byteorder.h>
#include <linux/types.h> #include <linux/types.h>
#include <asm/byteorder.h>
#include <asm/unaligned.h>
#define TGR192_DIGEST_SIZE 24 #define TGR192_DIGEST_SIZE 24
#define TGR160_DIGEST_SIZE 20 #define TGR160_DIGEST_SIZE 20
@ -468,10 +469,9 @@ static void tgr192_transform(struct tgr192_ctx *tctx, const u8 * data)
u64 a, b, c, aa, bb, cc; u64 a, b, c, aa, bb, cc;
u64 x[8]; u64 x[8];
int i; int i;
const __le64 *ptr = (const __le64 *)data;
for (i = 0; i < 8; i++) for (i = 0; i < 8; i++)
x[i] = le64_to_cpu(ptr[i]); x[i] = get_unaligned_le64(data + i * sizeof(__le64));
/* save */ /* save */
a = aa = tctx->a; a = aa = tctx->a;

11
drivers/bus/fsl-mc/fsl-mc-allocator.c
View File

@ -295,6 +295,14 @@ int __must_check fsl_mc_object_allocate(struct fsl_mc_device *mc_dev,
if (!mc_adev) if (!mc_adev)
goto error; goto error;
mc_adev->consumer_link = device_link_add(&mc_dev->dev,
&mc_adev->dev,
DL_FLAG_AUTOREMOVE_CONSUMER);
if (!mc_adev->consumer_link) {
error = -EINVAL;
goto error;
}
*new_mc_adev = mc_adev; *new_mc_adev = mc_adev;
return 0; return 0;
error: error:
@ -321,6 +329,9 @@ void fsl_mc_object_free(struct fsl_mc_device *mc_adev)
return; return;
fsl_mc_resource_free(resource); fsl_mc_resource_free(resource);
device_link_del(mc_adev->consumer_link);
mc_adev->consumer_link = NULL;
} }
EXPORT_SYMBOL_GPL(fsl_mc_object_free); EXPORT_SYMBOL_GPL(fsl_mc_object_free);

13
drivers/bus/fsl-mc/mc-io.c
View File

@ -209,9 +209,19 @@ int __must_check fsl_mc_portal_allocate(struct fsl_mc_device *mc_dev,
if (error < 0) if (error < 0)
goto error_cleanup_resource; goto error_cleanup_resource;
dpmcp_dev->consumer_link = device_link_add(&mc_dev->dev,
&dpmcp_dev->dev,
DL_FLAG_AUTOREMOVE_CONSUMER);
if (!dpmcp_dev->consumer_link) {
error = -EINVAL;
goto error_cleanup_mc_io;
}
*new_mc_io = mc_io; *new_mc_io = mc_io;
return 0; return 0;
error_cleanup_mc_io:
fsl_destroy_mc_io(mc_io);
error_cleanup_resource: error_cleanup_resource:
fsl_mc_resource_free(resource); fsl_mc_resource_free(resource);
return error; return error;
@ -244,6 +254,9 @@ void fsl_mc_portal_free(struct fsl_mc_io *mc_io)
fsl_destroy_mc_io(mc_io); fsl_destroy_mc_io(mc_io);
fsl_mc_resource_free(resource); fsl_mc_resource_free(resource);
device_link_del(dpmcp_dev->consumer_link);
dpmcp_dev->consumer_link = NULL;
} }
EXPORT_SYMBOL_GPL(fsl_mc_portal_free); EXPORT_SYMBOL_GPL(fsl_mc_portal_free);

16
drivers/char/hw_random/bcm2835-rng.c
View File

@ -168,14 +168,16 @@ static int bcm2835_rng_probe(struct platform_device *pdev)
priv->rng.read = bcm2835_rng_read; priv->rng.read = bcm2835_rng_read;
priv->rng.cleanup = bcm2835_rng_cleanup; priv->rng.cleanup = bcm2835_rng_cleanup;
rng_id = of_match_node(bcm2835_rng_of_match, np); if (dev_of_node(dev)) {
if (!rng_id) rng_id = of_match_node(bcm2835_rng_of_match, np);
return -EINVAL; if (!rng_id)
return -EINVAL;
/* Check for rng init function, execute it */ /* Check for rng init function, execute it */
of_data = rng_id->data; of_data = rng_id->data;
if (of_data) if (of_data)
priv->mask_interrupts = of_data->mask_interrupts; priv->mask_interrupts = of_data->mask_interrupts;
}
/* register driver */ /* register driver */
err = devm_hwrng_register(dev, &priv->rng); err = devm_hwrng_register(dev, &priv->rng);

2
drivers/char/hw_random/virtio-rng.c
View File

@ -73,7 +73,7 @@ static int virtio_read(struct hwrng *rng, void *buf, size_t size, bool wait)
if (!vi->busy) { if (!vi->busy) {
vi->busy = true; vi->busy = true;
init_completion(&vi->have_data); reinit_completion(&vi->have_data);
register_buffer(vi, buf, size); register_buffer(vi, buf, size);
} }

87
drivers/crypto/amcc/crypto4xx_core.c
View File

@ -40,9 +40,11 @@
#include <crypto/ctr.h> #include <crypto/ctr.h>
#include <crypto/gcm.h> #include <crypto/gcm.h>
#include <crypto/sha.h> #include <crypto/sha.h>
#include <crypto/rng.h>
#include <crypto/scatterwalk.h> #include <crypto/scatterwalk.h>
#include <crypto/skcipher.h> #include <crypto/skcipher.h>
#include <crypto/internal/aead.h> #include <crypto/internal/aead.h>
#include <crypto/internal/rng.h>
#include <crypto/internal/skcipher.h> #include <crypto/internal/skcipher.h>
#include "crypto4xx_reg_def.h" #include "crypto4xx_reg_def.h"
#include "crypto4xx_core.h" #include "crypto4xx_core.h"
@ -1035,6 +1037,10 @@ static int crypto4xx_register_alg(struct crypto4xx_device *sec_dev,
rc = crypto_register_ahash(&alg->alg.u.hash); rc = crypto_register_ahash(&alg->alg.u.hash);
break; break;
case CRYPTO_ALG_TYPE_RNG:
rc = crypto_register_rng(&alg->alg.u.rng);
break;
default: default:
rc = crypto_register_skcipher(&alg->alg.u.cipher); rc = crypto_register_skcipher(&alg->alg.u.cipher);
break; break;
@ -1064,6 +1070,10 @@ static void crypto4xx_unregister_alg(struct crypto4xx_device *sec_dev)
crypto_unregister_aead(&alg->alg.u.aead); crypto_unregister_aead(&alg->alg.u.aead);
break; break;
case CRYPTO_ALG_TYPE_RNG:
crypto_unregister_rng(&alg->alg.u.rng);
break;
default: default:
crypto_unregister_skcipher(&alg->alg.u.cipher); crypto_unregister_skcipher(&alg->alg.u.cipher);
} }
@ -1122,6 +1132,69 @@ static irqreturn_t crypto4xx_ce_interrupt_handler_revb(int irq, void *data)
PPC4XX_TMO_ERR_INT); PPC4XX_TMO_ERR_INT);
} }
static int ppc4xx_prng_data_read(struct crypto4xx_device *dev,
u8 *data, unsigned int max)
{
unsigned int i, curr = 0;
u32 val[2];
do {
/* trigger PRN generation */
writel(PPC4XX_PRNG_CTRL_AUTO_EN,
dev->ce_base + CRYPTO4XX_PRNG_CTRL);
for (i = 0; i < 1024; i++) {
/* usually 19 iterations are enough */
if ((readl(dev->ce_base + CRYPTO4XX_PRNG_STAT) &
CRYPTO4XX_PRNG_STAT_BUSY))
continue;
val[0] = readl_be(dev->ce_base + CRYPTO4XX_PRNG_RES_0);
val[1] = readl_be(dev->ce_base + CRYPTO4XX_PRNG_RES_1);
break;
}
if (i == 1024)
return -ETIMEDOUT;
if ((max - curr) >= 8) {
memcpy(data, &val, 8);
data += 8;
curr += 8;
} else {
/* copy only remaining bytes */
memcpy(data, &val, max - curr);
break;
}
} while (curr < max);
return curr;
}
static int crypto4xx_prng_generate(struct crypto_rng *tfm,
const u8 *src, unsigned int slen,
u8 *dstn, unsigned int dlen)
{
struct rng_alg *alg = crypto_rng_alg(tfm);
struct crypto4xx_alg *amcc_alg;
struct crypto4xx_device *dev;
int ret;
amcc_alg = container_of(alg, struct crypto4xx_alg, alg.u.rng);
dev = amcc_alg->dev;
mutex_lock(&dev->core_dev->rng_lock);
ret = ppc4xx_prng_data_read(dev, dstn, dlen);
mutex_unlock(&dev->core_dev->rng_lock);
return ret;
}
static int crypto4xx_prng_seed(struct crypto_rng *tfm, const u8 *seed,
unsigned int slen)
{
return 0;
}
/** /**
* Supported Crypto Algorithms * Supported Crypto Algorithms
*/ */
@ -1291,6 +1364,18 @@ static struct crypto4xx_alg_common crypto4xx_alg[] = {
.cra_module = THIS_MODULE, .cra_module = THIS_MODULE,
}, },
} }, } },
{ .type = CRYPTO_ALG_TYPE_RNG, .u.rng = {
.base = {
.cra_name = "stdrng",
.cra_driver_name = "crypto4xx_rng",
.cra_priority = 300,
.cra_ctxsize = 0,
.cra_module = THIS_MODULE,
},
.generate = crypto4xx_prng_generate,
.seed = crypto4xx_prng_seed,
.seedsize = 0,
} },
}; };
/** /**
@ -1360,6 +1445,7 @@ static int crypto4xx_probe(struct platform_device *ofdev)
core_dev->dev->core_dev = core_dev; core_dev->dev->core_dev = core_dev;
core_dev->dev->is_revb = is_revb; core_dev->dev->is_revb = is_revb;
core_dev->device = dev; core_dev->device = dev;
mutex_init(&core_dev->rng_lock);
spin_lock_init(&core_dev->lock); spin_lock_init(&core_dev->lock);
INIT_LIST_HEAD(&core_dev->dev->alg_list); INIT_LIST_HEAD(&core_dev->dev->alg_list);
ratelimit_default_init(&core_dev->dev->aead_ratelimit); ratelimit_default_init(&core_dev->dev->aead_ratelimit);
@ -1439,6 +1525,7 @@ static int crypto4xx_remove(struct platform_device *ofdev)
tasklet_kill(&core_dev->tasklet); tasklet_kill(&core_dev->tasklet);
/* Un-register with Linux CryptoAPI */ /* Un-register with Linux CryptoAPI */
crypto4xx_unregister_alg(core_dev->dev); crypto4xx_unregister_alg(core_dev->dev);
mutex_destroy(&core_dev->rng_lock);
/* Free all allocated memory */ /* Free all allocated memory */
crypto4xx_stop_all(core_dev); crypto4xx_stop_all(core_dev);

4
drivers/crypto/amcc/crypto4xx_core.h
View File

@ -23,8 +23,10 @@
#define __CRYPTO4XX_CORE_H__ #define __CRYPTO4XX_CORE_H__
#include <linux/ratelimit.h> #include <linux/ratelimit.h>
#include <linux/mutex.h>
#include <crypto/internal/hash.h> #include <crypto/internal/hash.h>
#include <crypto/internal/aead.h> #include <crypto/internal/aead.h>
#include <crypto/internal/rng.h>
#include <crypto/internal/skcipher.h> #include <crypto/internal/skcipher.h>
#include "crypto4xx_reg_def.h" #include "crypto4xx_reg_def.h"
#include "crypto4xx_sa.h" #include "crypto4xx_sa.h"
@ -119,6 +121,7 @@ struct crypto4xx_core_device {
u32 irq; u32 irq;
struct tasklet_struct tasklet; struct tasklet_struct tasklet;
spinlock_t lock; spinlock_t lock;
struct mutex rng_lock;
}; };
struct crypto4xx_ctx { struct crypto4xx_ctx {
@ -143,6 +146,7 @@ struct crypto4xx_alg_common {
struct skcipher_alg cipher; struct skcipher_alg cipher;
struct ahash_alg hash; struct ahash_alg hash;
struct aead_alg aead; struct aead_alg aead;
struct rng_alg rng;
} u; } u;
}; };

1
drivers/crypto/amcc/crypto4xx_reg_def.h
View File

@ -100,6 +100,7 @@
#define CRYPTO4XX_ENDIAN_CFG 0x000600d8 #define CRYPTO4XX_ENDIAN_CFG 0x000600d8
#define CRYPTO4XX_PRNG_STAT 0x00070000 #define CRYPTO4XX_PRNG_STAT 0x00070000
#define CRYPTO4XX_PRNG_STAT_BUSY 0x1
#define CRYPTO4XX_PRNG_CTRL 0x00070004 #define CRYPTO4XX_PRNG_CTRL 0x00070004
#define CRYPTO4XX_PRNG_SEED_L 0x00070008 #define CRYPTO4XX_PRNG_SEED_L 0x00070008
#define CRYPTO4XX_PRNG_SEED_H 0x0007000c #define CRYPTO4XX_PRNG_SEED_H 0x0007000c

4
drivers/crypto/amcc/crypto4xx_trng.c
View File

@ -80,8 +80,10 @@ void ppc4xx_trng_probe(struct crypto4xx_core_device *core_dev)
/* Find the TRNG device node and map it */ /* Find the TRNG device node and map it */
trng = of_find_matching_node(NULL, ppc4xx_trng_match); trng = of_find_matching_node(NULL, ppc4xx_trng_match);
if (!trng || !of_device_is_available(trng)) if (!trng || !of_device_is_available(trng)) {
of_node_put(trng);
return; return;
}
dev->trng_base = of_iomap(trng, 0); dev->trng_base = of_iomap(trng, 0);
of_node_put(trng); of_node_put(trng);

4
drivers/crypto/amcc/crypto4xx_trng.h
View File

@ -26,9 +26,9 @@ void ppc4xx_trng_probe(struct crypto4xx_core_device *core_dev);
void ppc4xx_trng_remove(struct crypto4xx_core_device *core_dev); void ppc4xx_trng_remove(struct crypto4xx_core_device *core_dev);
#else #else
static inline void ppc4xx_trng_probe( static inline void ppc4xx_trng_probe(
struct crypto4xx_device *dev __maybe_unused) { } struct crypto4xx_core_device *dev __maybe_unused) { }
static inline void ppc4xx_trng_remove( static inline void ppc4xx_trng_remove(
struct crypto4xx_device *dev __maybe_unused) { } struct crypto4xx_core_device *dev __maybe_unused) { }
#endif #endif
#endif #endif

2
drivers/crypto/atmel-tdes.c
View File

@ -785,7 +785,7 @@ static int atmel_des_setkey(struct crypto_ablkcipher *tfm, const u8 *key,
} }
err = des_ekey(tmp, key); err = des_ekey(tmp, key);
if (err == 0 && (ctfm->crt_flags & CRYPTO_TFM_REQ_WEAK_KEY)) { if (err == 0 && (ctfm->crt_flags & CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) {
ctfm->crt_flags |= CRYPTO_TFM_RES_WEAK_KEY; ctfm->crt_flags |= CRYPTO_TFM_RES_WEAK_KEY;
return -EINVAL; return -EINVAL;
} }

328
drivers/crypto/axis/artpec6_crypto.c
View File

@ -135,8 +135,6 @@
#define regk_crypto_ext 0x00000001 #define regk_crypto_ext 0x00000001
#define regk_crypto_hmac_sha1 0x00000007 #define regk_crypto_hmac_sha1 0x00000007
#define regk_crypto_hmac_sha256 0x00000009 #define regk_crypto_hmac_sha256 0x00000009
#define regk_crypto_hmac_sha384 0x0000000b
#define regk_crypto_hmac_sha512 0x0000000d
#define regk_crypto_init 0x00000000 #define regk_crypto_init 0x00000000
#define regk_crypto_key_128 0x00000000 #define regk_crypto_key_128 0x00000000
#define regk_crypto_key_192 0x00000001 #define regk_crypto_key_192 0x00000001
@ -144,8 +142,6 @@
#define regk_crypto_null 0x00000000 #define regk_crypto_null 0x00000000
#define regk_crypto_sha1 0x00000006 #define regk_crypto_sha1 0x00000006
#define regk_crypto_sha256 0x00000008 #define regk_crypto_sha256 0x00000008
#define regk_crypto_sha384 0x0000000a
#define regk_crypto_sha512 0x0000000c
/* DMA descriptor structures */ /* DMA descriptor structures */
struct pdma_descr_ctrl { struct pdma_descr_ctrl {
@ -190,8 +186,6 @@ struct pdma_stat_descr {
/* Hash modes (including HMAC variants) */ /* Hash modes (including HMAC variants) */
#define ARTPEC6_CRYPTO_HASH_SHA1 1 #define ARTPEC6_CRYPTO_HASH_SHA1 1
#define ARTPEC6_CRYPTO_HASH_SHA256 2 #define ARTPEC6_CRYPTO_HASH_SHA256 2
#define ARTPEC6_CRYPTO_HASH_SHA384 3
#define ARTPEC6_CRYPTO_HASH_SHA512 4
/* Crypto modes */ /* Crypto modes */
#define ARTPEC6_CRYPTO_CIPHER_AES_ECB 1 #define ARTPEC6_CRYPTO_CIPHER_AES_ECB 1
@ -284,6 +278,7 @@ enum artpec6_crypto_hash_flags {
struct artpec6_crypto_req_common { struct artpec6_crypto_req_common {
struct list_head list; struct list_head list;
struct list_head complete_in_progress;
struct artpec6_crypto_dma_descriptors *dma; struct artpec6_crypto_dma_descriptors *dma;
struct crypto_async_request *req; struct crypto_async_request *req;
void (*complete)(struct crypto_async_request *req); void (*complete)(struct crypto_async_request *req);
@ -291,11 +286,11 @@ struct artpec6_crypto_req_common {
}; };
struct artpec6_hash_request_context { struct artpec6_hash_request_context {
char partial_buffer[SHA512_BLOCK_SIZE]; char partial_buffer[SHA256_BLOCK_SIZE];
char partial_buffer_out[SHA512_BLOCK_SIZE]; char partial_buffer_out[SHA256_BLOCK_SIZE];
char key_buffer[SHA512_BLOCK_SIZE]; char key_buffer[SHA256_BLOCK_SIZE];
char pad_buffer[SHA512_BLOCK_SIZE + 32]; char pad_buffer[SHA256_BLOCK_SIZE + 32];
unsigned char digeststate[SHA512_DIGEST_SIZE]; unsigned char digeststate[SHA256_DIGEST_SIZE];
size_t partial_bytes; size_t partial_bytes;
u64 digcnt; u64 digcnt;
u32 key_md; u32 key_md;
@ -305,8 +300,8 @@ struct artpec6_hash_request_context {
}; };
struct artpec6_hash_export_state { struct artpec6_hash_export_state {
char partial_buffer[SHA512_BLOCK_SIZE]; char partial_buffer[SHA256_BLOCK_SIZE];
unsigned char digeststate[SHA512_DIGEST_SIZE]; unsigned char digeststate[SHA256_DIGEST_SIZE];
size_t partial_bytes; size_t partial_bytes;
u64 digcnt; u64 digcnt;
int oper; int oper;
@ -314,7 +309,7 @@ struct artpec6_hash_export_state {
}; };
struct artpec6_hashalg_context { struct artpec6_hashalg_context {
char hmac_key[SHA512_BLOCK_SIZE]; char hmac_key[SHA256_BLOCK_SIZE];
size_t hmac_key_length; size_t hmac_key_length;
struct crypto_shash *child_hash; struct crypto_shash *child_hash;
}; };
@ -670,8 +665,8 @@ artpec6_crypto_dma_map_descs(struct artpec6_crypto_req_common *common)
* to be written. * to be written.
*/ */
return artpec6_crypto_dma_map_single(common, return artpec6_crypto_dma_map_single(common,
dma->stat + dma->in_cnt - 1, dma->stat,
sizeof(dma->stat[0]), sizeof(dma->stat[0]) * dma->in_cnt,
DMA_BIDIRECTIONAL, DMA_BIDIRECTIONAL,
&dma->stat_dma_addr); &dma->stat_dma_addr);
} }
@ -1315,8 +1310,7 @@ static int artpec6_crypto_prepare_hash(struct ahash_request *areq)
struct artpec6_hashalg_context *ctx = crypto_tfm_ctx(areq->base.tfm); struct artpec6_hashalg_context *ctx = crypto_tfm_ctx(areq->base.tfm);
struct artpec6_hash_request_context *req_ctx = ahash_request_ctx(areq); struct artpec6_hash_request_context *req_ctx = ahash_request_ctx(areq);
size_t digestsize = crypto_ahash_digestsize(crypto_ahash_reqtfm(areq)); size_t digestsize = crypto_ahash_digestsize(crypto_ahash_reqtfm(areq));
size_t contextsize = digestsize == SHA384_DIGEST_SIZE ? size_t contextsize = digestsize;
SHA512_DIGEST_SIZE : digestsize;
size_t blocksize = crypto_tfm_alg_blocksize( size_t blocksize = crypto_tfm_alg_blocksize(
crypto_ahash_tfm(crypto_ahash_reqtfm(areq))); crypto_ahash_tfm(crypto_ahash_reqtfm(areq)));
struct artpec6_crypto_req_common *common = &req_ctx->common; struct artpec6_crypto_req_common *common = &req_ctx->common;
@ -1456,7 +1450,6 @@ static int artpec6_crypto_prepare_hash(struct ahash_request *areq)
/* Finalize */ /* Finalize */
if (req_ctx->hash_flags & HASH_FLAG_FINALIZE) { if (req_ctx->hash_flags & HASH_FLAG_FINALIZE) {
bool needtrim = contextsize != digestsize;
size_t hash_pad_len; size_t hash_pad_len;
u64 digest_bits; u64 digest_bits;
u32 oper; u32 oper;
@ -1502,19 +1495,10 @@ static int artpec6_crypto_prepare_hash(struct ahash_request *areq)
/* Descriptor for the final result */ /* Descriptor for the final result */
error = artpec6_crypto_setup_in_descr(common, areq->result, error = artpec6_crypto_setup_in_descr(common, areq->result,
digestsize, digestsize,
!needtrim); true);
if (error) if (error)
return error; return error;
if (needtrim) {
/* Discard the extra context bytes for SHA-384 */
error = artpec6_crypto_setup_in_descr(common,
req_ctx->partial_buffer,
digestsize - contextsize, true);
if (error)
return error;
}
} else { /* This is not the final operation for this request */ } else { /* This is not the final operation for this request */
if (!run_hw) if (!run_hw)
return ARTPEC6_CRYPTO_PREPARE_HASH_NO_START; return ARTPEC6_CRYPTO_PREPARE_HASH_NO_START;
@ -1923,7 +1907,7 @@ static int artpec6_crypto_prepare_aead(struct aead_request *areq)
/* For the decryption, cryptlen includes the tag. */ /* For the decryption, cryptlen includes the tag. */
input_length = areq->cryptlen; input_length = areq->cryptlen;
if (req_ctx->decrypt) if (req_ctx->decrypt)
input_length -= AES_BLOCK_SIZE; input_length -= crypto_aead_authsize(cipher);
/* Prepare the context buffer */ /* Prepare the context buffer */
req_ctx->hw_ctx.aad_length_bits = req_ctx->hw_ctx.aad_length_bits =
@ -1988,7 +1972,7 @@ static int artpec6_crypto_prepare_aead(struct aead_request *areq)
size_t output_len = areq->cryptlen; size_t output_len = areq->cryptlen;
if (req_ctx->decrypt) if (req_ctx->decrypt)
output_len -= AES_BLOCK_SIZE; output_len -= crypto_aead_authsize(cipher);
artpec6_crypto_walk_init(&walk, areq->dst); artpec6_crypto_walk_init(&walk, areq->dst);
@ -2017,19 +2001,32 @@ static int artpec6_crypto_prepare_aead(struct aead_request *areq)
* the output ciphertext. For decryption it is put in a context * the output ciphertext. For decryption it is put in a context
* buffer for later compare against the input tag. * buffer for later compare against the input tag.
*/ */
count = AES_BLOCK_SIZE;
if (req_ctx->decrypt) { if (req_ctx->decrypt) {
ret = artpec6_crypto_setup_in_descr(common, ret = artpec6_crypto_setup_in_descr(common,
req_ctx->decryption_tag, count, false); req_ctx->decryption_tag, AES_BLOCK_SIZE, false);
if (ret) if (ret)
return ret; return ret;
} else { } else {
/* For encryption the requested tag size may be smaller
* than the hardware's generated tag.
*/
size_t authsize = crypto_aead_authsize(cipher);
ret = artpec6_crypto_setup_sg_descrs_in(common, &walk, ret = artpec6_crypto_setup_sg_descrs_in(common, &walk,
count); authsize);
if (ret) if (ret)
return ret; return ret;
if (authsize < AES_BLOCK_SIZE) {
count = AES_BLOCK_SIZE - authsize;
ret = artpec6_crypto_setup_in_descr(common,
ac->pad_buffer,
count, false);
if (ret)
return ret;
}
} }
} }
@ -2045,7 +2042,8 @@ static int artpec6_crypto_prepare_aead(struct aead_request *areq)
return artpec6_crypto_dma_map_descs(common); return artpec6_crypto_dma_map_descs(common);
} }
static void artpec6_crypto_process_queue(struct artpec6_crypto *ac) static void artpec6_crypto_process_queue(struct artpec6_crypto *ac,
struct list_head *completions)
{ {
struct artpec6_crypto_req_common *req; struct artpec6_crypto_req_common *req;
@ -2056,7 +2054,7 @@ static void artpec6_crypto_process_queue(struct artpec6_crypto *ac)
list_move_tail(&req->list, &ac->pending); list_move_tail(&req->list, &ac->pending);
artpec6_crypto_start_dma(req); artpec6_crypto_start_dma(req);
req->req->complete(req->req, -EINPROGRESS); list_add_tail(&req->complete_in_progress, completions);
} }
/* /*
@ -2086,6 +2084,11 @@ static void artpec6_crypto_task(unsigned long data)
struct artpec6_crypto *ac = (struct artpec6_crypto *)data; struct artpec6_crypto *ac = (struct artpec6_crypto *)data;
struct artpec6_crypto_req_common *req; struct artpec6_crypto_req_common *req;
struct artpec6_crypto_req_common *n; struct artpec6_crypto_req_common *n;
struct list_head complete_done;
struct list_head complete_in_progress;
INIT_LIST_HEAD(&complete_done);
INIT_LIST_HEAD(&complete_in_progress);
if (list_empty(&ac->pending)) { if (list_empty(&ac->pending)) {
pr_debug("Spurious IRQ\n"); pr_debug("Spurious IRQ\n");
@ -2097,9 +2100,12 @@ static void artpec6_crypto_task(unsigned long data)
list_for_each_entry_safe(req, n, &ac->pending, list) { list_for_each_entry_safe(req, n, &ac->pending, list) {
struct artpec6_crypto_dma_descriptors *dma = req->dma; struct artpec6_crypto_dma_descriptors *dma = req->dma;
u32 stat; u32 stat;
dma_addr_t stataddr;
dma_sync_single_for_cpu(artpec6_crypto_dev, dma->stat_dma_addr, stataddr = dma->stat_dma_addr + 4 * (req->dma->in_cnt - 1);
sizeof(dma->stat[0]), dma_sync_single_for_cpu(artpec6_crypto_dev,
stataddr,
4,
DMA_BIDIRECTIONAL); DMA_BIDIRECTIONAL);
stat = req->dma->stat[req->dma->in_cnt-1]; stat = req->dma->stat[req->dma->in_cnt-1];
@ -2119,19 +2125,30 @@ static void artpec6_crypto_task(unsigned long data)
pr_debug("Completing request %p\n", req); pr_debug("Completing request %p\n", req);
list_del(&req->list); list_move_tail(&req->list, &complete_done);
artpec6_crypto_dma_unmap_all(req);
artpec6_crypto_copy_bounce_buffers(req);
ac->pending_count--; ac->pending_count--;
}
artpec6_crypto_process_queue(ac, &complete_in_progress);
spin_unlock_bh(&ac->queue_lock);
/* Perform the completion callbacks without holding the queue lock
* to allow new request submissions from the callbacks.
*/
list_for_each_entry_safe(req, n, &complete_done, list) {
artpec6_crypto_dma_unmap_all(req);
artpec6_crypto_copy_bounce_buffers(req);
artpec6_crypto_common_destroy(req); artpec6_crypto_common_destroy(req);
req->complete(req->req); req->complete(req->req);
} }
artpec6_crypto_process_queue(ac); list_for_each_entry_safe(req, n, &complete_in_progress,
complete_in_progress) {
spin_unlock_bh(&ac->queue_lock); req->req->complete(req->req, -EINPROGRESS);
}
} }
static void artpec6_crypto_complete_crypto(struct crypto_async_request *req) static void artpec6_crypto_complete_crypto(struct crypto_async_request *req)
@ -2170,27 +2187,29 @@ static void artpec6_crypto_complete_aead(struct crypto_async_request *req)
/* Verify GCM hashtag. */ /* Verify GCM hashtag. */
struct aead_request *areq = container_of(req, struct aead_request *areq = container_of(req,
struct aead_request, base); struct aead_request, base);
struct crypto_aead *aead = crypto_aead_reqtfm(areq);
struct artpec6_crypto_aead_req_ctx *req_ctx = aead_request_ctx(areq); struct artpec6_crypto_aead_req_ctx *req_ctx = aead_request_ctx(areq);
if (req_ctx->decrypt) { if (req_ctx->decrypt) {
u8 input_tag[AES_BLOCK_SIZE]; u8 input_tag[AES_BLOCK_SIZE];
unsigned int authsize = crypto_aead_authsize(aead);
sg_pcopy_to_buffer(areq->src, sg_pcopy_to_buffer(areq->src,
sg_nents(areq->src), sg_nents(areq->src),
input_tag, input_tag,
AES_BLOCK_SIZE, authsize,
areq->assoclen + areq->cryptlen - areq->assoclen + areq->cryptlen -
AES_BLOCK_SIZE); authsize);
if (memcmp(req_ctx->decryption_tag, if (crypto_memneq(req_ctx->decryption_tag,
input_tag, input_tag,
AES_BLOCK_SIZE)) { authsize)) {
pr_debug("***EBADMSG:\n"); pr_debug("***EBADMSG:\n");
print_hex_dump_debug("ref:", DUMP_PREFIX_ADDRESS, 32, 1, print_hex_dump_debug("ref:", DUMP_PREFIX_ADDRESS, 32, 1,
input_tag, AES_BLOCK_SIZE, true); input_tag, authsize, true);
print_hex_dump_debug("out:", DUMP_PREFIX_ADDRESS, 32, 1, print_hex_dump_debug("out:", DUMP_PREFIX_ADDRESS, 32, 1,
req_ctx->decryption_tag, req_ctx->decryption_tag,
AES_BLOCK_SIZE, true); authsize, true);
result = -EBADMSG; result = -EBADMSG;
} }
@ -2266,13 +2285,6 @@ artpec6_crypto_init_hash(struct ahash_request *req, u8 type, int hmac)
case ARTPEC6_CRYPTO_HASH_SHA256: case ARTPEC6_CRYPTO_HASH_SHA256:
oper = hmac ? regk_crypto_hmac_sha256 : regk_crypto_sha256; oper = hmac ? regk_crypto_hmac_sha256 : regk_crypto_sha256;
break; break;
case ARTPEC6_CRYPTO_HASH_SHA384:
oper = hmac ? regk_crypto_hmac_sha384 : regk_crypto_sha384;
break;
case ARTPEC6_CRYPTO_HASH_SHA512:
oper = hmac ? regk_crypto_hmac_sha512 : regk_crypto_sha512;
break;
default: default:
pr_err("%s: Unsupported hash type 0x%x\n", MODULE_NAME, type); pr_err("%s: Unsupported hash type 0x%x\n", MODULE_NAME, type);
return -EINVAL; return -EINVAL;
@ -2368,53 +2380,11 @@ static int artpec6_crypto_sha256_digest(struct ahash_request *req)
return artpec6_crypto_prepare_submit_hash(req); return artpec6_crypto_prepare_submit_hash(req);
} }
static int __maybe_unused artpec6_crypto_sha384_init(struct ahash_request *req)
{
return artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA384, 0);
}
static int __maybe_unused
artpec6_crypto_sha384_digest(struct ahash_request *req)
{
struct artpec6_hash_request_context *req_ctx = ahash_request_ctx(req);
artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA384, 0);
req_ctx->hash_flags |= HASH_FLAG_UPDATE | HASH_FLAG_FINALIZE;
return artpec6_crypto_prepare_submit_hash(req);
}
static int artpec6_crypto_sha512_init(struct ahash_request *req)
{
return artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA512, 0);
}
static int artpec6_crypto_sha512_digest(struct ahash_request *req)
{
struct artpec6_hash_request_context *req_ctx = ahash_request_ctx(req);
artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA512, 0);
req_ctx->hash_flags |= HASH_FLAG_UPDATE | HASH_FLAG_FINALIZE;
return artpec6_crypto_prepare_submit_hash(req);
}
static int artpec6_crypto_hmac_sha256_init(struct ahash_request *req) static int artpec6_crypto_hmac_sha256_init(struct ahash_request *req)
{ {
return artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA256, 1); return artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA256, 1);
} }
static int __maybe_unused
artpec6_crypto_hmac_sha384_init(struct ahash_request *req)
{
return artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA384, 1);
}
static int artpec6_crypto_hmac_sha512_init(struct ahash_request *req)
{
return artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA512, 1);
}
static int artpec6_crypto_hmac_sha256_digest(struct ahash_request *req) static int artpec6_crypto_hmac_sha256_digest(struct ahash_request *req)
{ {
struct artpec6_hash_request_context *req_ctx = ahash_request_ctx(req); struct artpec6_hash_request_context *req_ctx = ahash_request_ctx(req);
@ -2425,27 +2395,6 @@ static int artpec6_crypto_hmac_sha256_digest(struct ahash_request *req)
return artpec6_crypto_prepare_submit_hash(req); return artpec6_crypto_prepare_submit_hash(req);
} }
static int __maybe_unused
artpec6_crypto_hmac_sha384_digest(struct ahash_request *req)
{
struct artpec6_hash_request_context *req_ctx = ahash_request_ctx(req);
artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA384, 1);
req_ctx->hash_flags |= HASH_FLAG_UPDATE | HASH_FLAG_FINALIZE;
return artpec6_crypto_prepare_submit_hash(req);
}
static int artpec6_crypto_hmac_sha512_digest(struct ahash_request *req)
{
struct artpec6_hash_request_context *req_ctx = ahash_request_ctx(req);
artpec6_crypto_init_hash(req, ARTPEC6_CRYPTO_HASH_SHA512, 1);
req_ctx->hash_flags |= HASH_FLAG_UPDATE | HASH_FLAG_FINALIZE;
return artpec6_crypto_prepare_submit_hash(req);
}
static int artpec6_crypto_ahash_init_common(struct crypto_tfm *tfm, static int artpec6_crypto_ahash_init_common(struct crypto_tfm *tfm,
const char *base_hash_name) const char *base_hash_name)
{ {
@ -2480,17 +2429,6 @@ static int artpec6_crypto_ahash_init_hmac_sha256(struct crypto_tfm *tfm)
return artpec6_crypto_ahash_init_common(tfm, "sha256"); return artpec6_crypto_ahash_init_common(tfm, "sha256");
} }
static int __maybe_unused
artpec6_crypto_ahash_init_hmac_sha384(struct crypto_tfm *tfm)
{
return artpec6_crypto_ahash_init_common(tfm, "sha384");
}
static int artpec6_crypto_ahash_init_hmac_sha512(struct crypto_tfm *tfm)
{
return artpec6_crypto_ahash_init_common(tfm, "sha512");
}
static void artpec6_crypto_ahash_exit(struct crypto_tfm *tfm) static void artpec6_crypto_ahash_exit(struct crypto_tfm *tfm)
{ {
struct artpec6_hashalg_context *tfm_ctx = crypto_tfm_ctx(tfm); struct artpec6_hashalg_context *tfm_ctx = crypto_tfm_ctx(tfm);
@ -2761,103 +2699,6 @@ static struct ahash_alg hash_algos[] = {
}, },
}; };
static struct ahash_alg artpec7_hash_algos[] = {
/* SHA-384 */
{
.init = artpec6_crypto_sha384_init,
.update = artpec6_crypto_hash_update,
.final = artpec6_crypto_hash_final,
.digest = artpec6_crypto_sha384_digest,
.import = artpec6_crypto_hash_import,
.export = artpec6_crypto_hash_export,
.halg.digestsize = SHA384_DIGEST_SIZE,
.halg.statesize = sizeof(struct artpec6_hash_export_state),
.halg.base = {
.cra_name = "sha384",
.cra_driver_name = "artpec-sha384",
.cra_priority = 300,
.cra_flags = CRYPTO_ALG_ASYNC,
.cra_blocksize = SHA384_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct artpec6_hashalg_context),
.cra_alignmask = 3,
.cra_module = THIS_MODULE,
.cra_init = artpec6_crypto_ahash_init,
.cra_exit = artpec6_crypto_ahash_exit,
}
},
/* HMAC SHA-384 */
{
.init = artpec6_crypto_hmac_sha384_init,
.update = artpec6_crypto_hash_update,
.final = artpec6_crypto_hash_final,
.digest = artpec6_crypto_hmac_sha384_digest,
.import = artpec6_crypto_hash_import,
.export = artpec6_crypto_hash_export,
.setkey = artpec6_crypto_hash_set_key,
.halg.digestsize = SHA384_DIGEST_SIZE,
.halg.statesize = sizeof(struct artpec6_hash_export_state),
.halg.base = {
.cra_name = "hmac(sha384)",
.cra_driver_name = "artpec-hmac-sha384",
.cra_priority = 300,
.cra_flags = CRYPTO_ALG_ASYNC,
.cra_blocksize = SHA384_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct artpec6_hashalg_context),
.cra_alignmask = 3,
.cra_module = THIS_MODULE,
.cra_init = artpec6_crypto_ahash_init_hmac_sha384,
.cra_exit = artpec6_crypto_ahash_exit,
}
},
/* SHA-512 */
{
.init = artpec6_crypto_sha512_init,
.update = artpec6_crypto_hash_update,
.final = artpec6_crypto_hash_final,
.digest = artpec6_crypto_sha512_digest,
.import = artpec6_crypto_hash_import,
.export = artpec6_crypto_hash_export,
.halg.digestsize = SHA512_DIGEST_SIZE,
.halg.statesize = sizeof(struct artpec6_hash_export_state),
.halg.base = {
.cra_name = "sha512",
.cra_driver_name = "artpec-sha512",
.cra_priority = 300,
.cra_flags = CRYPTO_ALG_ASYNC,
.cra_blocksize = SHA512_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct artpec6_hashalg_context),
.cra_alignmask = 3,
.cra_module = THIS_MODULE,
.cra_init = artpec6_crypto_ahash_init,
.cra_exit = artpec6_crypto_ahash_exit,
}
},
/* HMAC SHA-512 */
{
.init = artpec6_crypto_hmac_sha512_init,
.update = artpec6_crypto_hash_update,
.final = artpec6_crypto_hash_final,
.digest = artpec6_crypto_hmac_sha512_digest,
.import = artpec6_crypto_hash_import,
.export = artpec6_crypto_hash_export,
.setkey = artpec6_crypto_hash_set_key,
.halg.digestsize = SHA512_DIGEST_SIZE,
.halg.statesize = sizeof(struct artpec6_hash_export_state),
.halg.base = {
.cra_name = "hmac(sha512)",
.cra_driver_name = "artpec-hmac-sha512",
.cra_priority = 300,
.cra_flags = CRYPTO_ALG_ASYNC,
.cra_blocksize = SHA512_BLOCK_SIZE,
.cra_ctxsize = sizeof(struct artpec6_hashalg_context),
.cra_alignmask = 3,
.cra_module = THIS_MODULE,
.cra_init = artpec6_crypto_ahash_init_hmac_sha512,
.cra_exit = artpec6_crypto_ahash_exit,
}
},
};
/* Crypto */ /* Crypto */
static struct skcipher_alg crypto_algos[] = { static struct skcipher_alg crypto_algos[] = {
/* AES - ECB */ /* AES - ECB */
@ -2984,12 +2825,6 @@ static void artpec6_crypto_init_debugfs(void)
{ {
dbgfs_root = debugfs_create_dir("artpec6_crypto", NULL); dbgfs_root = debugfs_create_dir("artpec6_crypto", NULL);
if (!dbgfs_root || IS_ERR(dbgfs_root)) {
dbgfs_root = NULL;
pr_err("%s: Could not initialise debugfs!\n", MODULE_NAME);
return;
}
#ifdef CONFIG_FAULT_INJECTION #ifdef CONFIG_FAULT_INJECTION
fault_create_debugfs_attr("fail_status_read", dbgfs_root, fault_create_debugfs_attr("fail_status_read", dbgfs_root,
&artpec6_crypto_fail_status_read); &artpec6_crypto_fail_status_read);
@ -3001,9 +2836,6 @@ static void artpec6_crypto_init_debugfs(void)
static void artpec6_crypto_free_debugfs(void) static void artpec6_crypto_free_debugfs(void)
{ {
if (!dbgfs_root)
return;
debugfs_remove_recursive(dbgfs_root); debugfs_remove_recursive(dbgfs_root);
dbgfs_root = NULL; dbgfs_root = NULL;
} }
@ -3104,19 +2936,10 @@ static int artpec6_crypto_probe(struct platform_device *pdev)
goto disable_hw; goto disable_hw;
} }
if (variant != ARTPEC6_CRYPTO) {
err = crypto_register_ahashes(artpec7_hash_algos,
ARRAY_SIZE(artpec7_hash_algos));
if (err) {
dev_err(dev, "Failed to register ahashes\n");
goto unregister_ahashes;
}
}
err = crypto_register_skciphers(crypto_algos, ARRAY_SIZE(crypto_algos)); err = crypto_register_skciphers(crypto_algos, ARRAY_SIZE(crypto_algos));
if (err) { if (err) {
dev_err(dev, "Failed to register ciphers\n"); dev_err(dev, "Failed to register ciphers\n");
goto unregister_a7_ahashes; goto unregister_ahashes;
} }
err = crypto_register_aeads(aead_algos, ARRAY_SIZE(aead_algos)); err = crypto_register_aeads(aead_algos, ARRAY_SIZE(aead_algos));
@ -3129,10 +2952,6 @@ static int artpec6_crypto_probe(struct platform_device *pdev)
unregister_algs: unregister_algs:
crypto_unregister_skciphers(crypto_algos, ARRAY_SIZE(crypto_algos)); crypto_unregister_skciphers(crypto_algos, ARRAY_SIZE(crypto_algos));
unregister_a7_ahashes:
if (variant != ARTPEC6_CRYPTO)
crypto_unregister_ahashes(artpec7_hash_algos,
ARRAY_SIZE(artpec7_hash_algos));
unregister_ahashes: unregister_ahashes:
crypto_unregister_ahashes(hash_algos, ARRAY_SIZE(hash_algos)); crypto_unregister_ahashes(hash_algos, ARRAY_SIZE(hash_algos));
disable_hw: disable_hw:
@ -3148,9 +2967,6 @@ static int artpec6_crypto_remove(struct platform_device *pdev)
int irq = platform_get_irq(pdev, 0); int irq = platform_get_irq(pdev, 0);
crypto_unregister_ahashes(hash_algos, ARRAY_SIZE(hash_algos)); crypto_unregister_ahashes(hash_algos, ARRAY_SIZE(hash_algos));
if (ac->variant != ARTPEC6_CRYPTO)
crypto_unregister_ahashes(artpec7_hash_algos,
ARRAY_SIZE(artpec7_hash_algos));
crypto_unregister_skciphers(crypto_algos, ARRAY_SIZE(crypto_algos)); crypto_unregister_skciphers(crypto_algos, ARRAY_SIZE(crypto_algos));
crypto_unregister_aeads(aead_algos, ARRAY_SIZE(aead_algos)); crypto_unregister_aeads(aead_algos, ARRAY_SIZE(aead_algos));

2
drivers/crypto/bcm/Makefile
View File

@ -11,5 +11,3 @@
obj-$(CONFIG_CRYPTO_DEV_BCM_SPU) := bcm_crypto_spu.o obj-$(CONFIG_CRYPTO_DEV_BCM_SPU) := bcm_crypto_spu.o
bcm_crypto_spu-objs := util.o spu.o spu2.o cipher.o bcm_crypto_spu-objs := util.o spu.o spu2.o cipher.o
ccflags-y += -I. -DBCMDRIVER

10
drivers/crypto/bcm/cipher.c
View File

@ -717,7 +717,7 @@ static int handle_ahash_req(struct iproc_reqctx_s *rctx)
*/ */
unsigned int new_data_len; unsigned int new_data_len;
unsigned int chunk_start = 0; unsigned int __maybe_unused chunk_start = 0;
u32 db_size; /* Length of data field, incl gcm and hash padding */ u32 db_size; /* Length of data field, incl gcm and hash padding */
int pad_len = 0; /* total pad len, including gcm, hash, stat padding */ int pad_len = 0; /* total pad len, including gcm, hash, stat padding */
u32 data_pad_len = 0; /* length of GCM/CCM padding */ u32 data_pad_len = 0; /* length of GCM/CCM padding */
@ -1675,8 +1675,6 @@ static void spu_rx_callback(struct mbox_client *cl, void *msg)
struct spu_hw *spu = &iproc_priv.spu; struct spu_hw *spu = &iproc_priv.spu;
struct brcm_message *mssg = msg; struct brcm_message *mssg = msg;
struct iproc_reqctx_s *rctx; struct iproc_reqctx_s *rctx;
struct iproc_ctx_s *ctx;
struct crypto_async_request *areq;
int err = 0; int err = 0;
rctx = mssg->ctx; rctx = mssg->ctx;
@ -1686,8 +1684,6 @@ static void spu_rx_callback(struct mbox_client *cl, void *msg)
err = -EFAULT; err = -EFAULT;
goto cb_finish; goto cb_finish;
} }
areq = rctx->parent;
ctx = rctx->ctx;
/* process the SPU status */ /* process the SPU status */
err = spu->spu_status_process(rctx->msg_buf.rx_stat); err = spu->spu_status_process(rctx->msg_buf.rx_stat);
@ -1822,7 +1818,7 @@ static int des_setkey(struct crypto_ablkcipher *cipher, const u8 *key,
if (keylen == DES_KEY_SIZE) { if (keylen == DES_KEY_SIZE) {
if (des_ekey(tmp, key) == 0) { if (des_ekey(tmp, key) == 0) {
if (crypto_ablkcipher_get_flags(cipher) & if (crypto_ablkcipher_get_flags(cipher) &
CRYPTO_TFM_REQ_WEAK_KEY) { CRYPTO_TFM_REQ_FORBID_WEAK_KEYS) {
u32 flags = CRYPTO_TFM_RES_WEAK_KEY; u32 flags = CRYPTO_TFM_RES_WEAK_KEY;
crypto_ablkcipher_set_flags(cipher, flags); crypto_ablkcipher_set_flags(cipher, flags);
@ -2876,7 +2872,7 @@ static int aead_authenc_setkey(struct crypto_aead *cipher,
if (des_ekey(tmp, keys.enckey) == 0) { if (des_ekey(tmp, keys.enckey) == 0) {
if (crypto_aead_get_flags(cipher) & if (crypto_aead_get_flags(cipher) &
CRYPTO_TFM_REQ_WEAK_KEY) { CRYPTO_TFM_REQ_FORBID_WEAK_KEYS) {
crypto_aead_set_flags(cipher, flags); crypto_aead_set_flags(cipher, flags);
return -EINVAL; return -EINVAL;
} }

4
drivers/crypto/bcm/cipher.h
View File

@ -23,6 +23,7 @@
#include <crypto/aes.h> #include <crypto/aes.h>
#include <crypto/internal/hash.h> #include <crypto/internal/hash.h>
#include <crypto/aead.h> #include <crypto/aead.h>
#include <crypto/arc4.h>
#include <crypto/gcm.h> #include <crypto/gcm.h>
#include <crypto/sha.h> #include <crypto/sha.h>
#include <crypto/sha3.h> #include <crypto/sha3.h>
@ -34,9 +35,6 @@
/* Driver supports up to MAX_SPUS SPU blocks */ /* Driver supports up to MAX_SPUS SPU blocks */
#define MAX_SPUS 16 #define MAX_SPUS 16
#define ARC4_MIN_KEY_SIZE 1
#define ARC4_MAX_KEY_SIZE 256
#define ARC4_BLOCK_SIZE 1
#define ARC4_STATE_SIZE 4 #define ARC4_STATE_SIZE 4
#define CCM_AES_IV_SIZE 16 #define CCM_AES_IV_SIZE 16

40
drivers/crypto/bcm/util.c
View File

@ -201,46 +201,6 @@ struct sdesc {
char ctx[]; char ctx[];
}; };
/* do a synchronous decrypt operation */
int do_decrypt(char *alg_name,
void *key_ptr, unsigned int key_len,
void *iv_ptr, void *src_ptr, void *dst_ptr,
unsigned int block_len)
{
struct scatterlist sg_in[1], sg_out[1];
struct crypto_blkcipher *tfm =
crypto_alloc_blkcipher(alg_name, 0, CRYPTO_ALG_ASYNC);
struct blkcipher_desc desc = {.tfm = tfm, .flags = 0 };
int ret = 0;
void *iv;
int ivsize;
flow_log("%s() name:%s block_len:%u\n", __func__, alg_name, block_len);
if (IS_ERR(tfm))
return PTR_ERR(tfm);
crypto_blkcipher_setkey((void *)tfm, key_ptr, key_len);
sg_init_table(sg_in, 1);
sg_set_buf(sg_in, src_ptr, block_len);
sg_init_table(sg_out, 1);
sg_set_buf(sg_out, dst_ptr, block_len);
iv = crypto_blkcipher_crt(tfm)->iv;
ivsize = crypto_blkcipher_ivsize(tfm);
memcpy(iv, iv_ptr, ivsize);
ret = crypto_blkcipher_decrypt(&desc, sg_out, sg_in, block_len);
crypto_free_blkcipher(tfm);
if (ret < 0)
pr_err("aes_decrypt failed %d\n", ret);
return ret;
}
/** /**
* do_shash() - Do a synchronous hash operation in software * do_shash() - Do a synchronous hash operation in software
* @name: The name of the hash algorithm * @name: The name of the hash algorithm

6
drivers/crypto/bcm/util.h
View File

@ -95,12 +95,6 @@ u32 spu_msg_sg_add(struct scatterlist **to_sg,
void add_to_ctr(u8 *ctr_pos, unsigned int increment); void add_to_ctr(u8 *ctr_pos, unsigned int increment);
/* do a synchronous decrypt operation */
int do_decrypt(char *alg_name,
void *key_ptr, unsigned int key_len,
void *iv_ptr, void *src_ptr, void *dst_ptr,
unsigned int block_len);
/* produce a message digest from data of length n bytes */ /* produce a message digest from data of length n bytes */
int do_shash(unsigned char *name, unsigned char *result, int do_shash(unsigned char *name, unsigned char *result,
const u8 *data1, unsigned int data1_len, const u8 *data1, unsigned int data1_len,

1
drivers/crypto/caam/Kconfig
View File

@ -91,6 +91,7 @@ config CRYPTO_DEV_FSL_CAAM_CRYPTO_API
select CRYPTO_AEAD select CRYPTO_AEAD
select CRYPTO_AUTHENC select CRYPTO_AUTHENC
select CRYPTO_BLKCIPHER select CRYPTO_BLKCIPHER
select CRYPTO_DES
help help
Selecting this will offload crypto for users of the Selecting this will offload crypto for users of the
scatterlist crypto API (such as the linux native IPSec scatterlist crypto API (such as the linux native IPSec

238
drivers/crypto/caam/caamalg.c
View File

@ -3,7 +3,7 @@
* caam - Freescale FSL CAAM support for crypto API * caam - Freescale FSL CAAM support for crypto API
* *
* Copyright 2008-2011 Freescale Semiconductor, Inc. * Copyright 2008-2011 Freescale Semiconductor, Inc.
* Copyright 2016-2018 NXP * Copyright 2016-2019 NXP
* *
* Based on talitos crypto API driver. * Based on talitos crypto API driver.
* *
@ -766,6 +766,27 @@ static int skcipher_setkey(struct crypto_skcipher *skcipher, const u8 *key,
return 0; return 0;
} }
static int des_skcipher_setkey(struct crypto_skcipher *skcipher,
const u8 *key, unsigned int keylen)
{
u32 tmp[DES3_EDE_EXPKEY_WORDS];
struct crypto_tfm *tfm = crypto_skcipher_tfm(skcipher);
if (keylen == DES3_EDE_KEY_SIZE &&
__des3_ede_setkey(tmp, &tfm->crt_flags, key, DES3_EDE_KEY_SIZE)) {
return -EINVAL;
}
if (!des_ekey(tmp, key) && (crypto_skcipher_get_flags(skcipher) &
CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) {
crypto_skcipher_set_flags(skcipher,
CRYPTO_TFM_RES_WEAK_KEY);
return -EINVAL;
}
return skcipher_setkey(skcipher, key, keylen);
}
static int xts_skcipher_setkey(struct crypto_skcipher *skcipher, const u8 *key, static int xts_skcipher_setkey(struct crypto_skcipher *skcipher, const u8 *key,
unsigned int keylen) unsigned int keylen)
{ {
@ -802,6 +823,8 @@ static int xts_skcipher_setkey(struct crypto_skcipher *skcipher, const u8 *key,
* aead_edesc - s/w-extended aead descriptor * aead_edesc - s/w-extended aead descriptor
* @src_nents: number of segments in input s/w scatterlist * @src_nents: number of segments in input s/w scatterlist
* @dst_nents: number of segments in output s/w scatterlist * @dst_nents: number of segments in output s/w scatterlist
* @mapped_src_nents: number of segments in input h/w link table
* @mapped_dst_nents: number of segments in output h/w link table
* @sec4_sg_bytes: length of dma mapped sec4_sg space * @sec4_sg_bytes: length of dma mapped sec4_sg space
* @sec4_sg_dma: bus physical mapped address of h/w link table * @sec4_sg_dma: bus physical mapped address of h/w link table
* @sec4_sg: pointer to h/w link table * @sec4_sg: pointer to h/w link table
@ -810,6 +833,8 @@ static int xts_skcipher_setkey(struct crypto_skcipher *skcipher, const u8 *key,
struct aead_edesc { struct aead_edesc {
int src_nents; int src_nents;
int dst_nents; int dst_nents;
int mapped_src_nents;
int mapped_dst_nents;
int sec4_sg_bytes; int sec4_sg_bytes;
dma_addr_t sec4_sg_dma; dma_addr_t sec4_sg_dma;
struct sec4_sg_entry *sec4_sg; struct sec4_sg_entry *sec4_sg;
@ -820,6 +845,8 @@ struct aead_edesc {
* skcipher_edesc - s/w-extended skcipher descriptor * skcipher_edesc - s/w-extended skcipher descriptor
* @src_nents: number of segments in input s/w scatterlist * @src_nents: number of segments in input s/w scatterlist
* @dst_nents: number of segments in output s/w scatterlist * @dst_nents: number of segments in output s/w scatterlist
* @mapped_src_nents: number of segments in input h/w link table
* @mapped_dst_nents: number of segments in output h/w link table
* @iv_dma: dma address of iv for checking continuity and link table * @iv_dma: dma address of iv for checking continuity and link table
* @sec4_sg_bytes: length of dma mapped sec4_sg space * @sec4_sg_bytes: length of dma mapped sec4_sg space
* @sec4_sg_dma: bus physical mapped address of h/w link table * @sec4_sg_dma: bus physical mapped address of h/w link table
@ -830,6 +857,8 @@ struct aead_edesc {
struct skcipher_edesc { struct skcipher_edesc {
int src_nents; int src_nents;
int dst_nents; int dst_nents;
int mapped_src_nents;
int mapped_dst_nents;
dma_addr_t iv_dma; dma_addr_t iv_dma;
int sec4_sg_bytes; int sec4_sg_bytes;
dma_addr_t sec4_sg_dma; dma_addr_t sec4_sg_dma;
@ -846,7 +875,8 @@ static void caam_unmap(struct device *dev, struct scatterlist *src,
if (dst != src) { if (dst != src) {
if (src_nents) if (src_nents)
dma_unmap_sg(dev, src, src_nents, DMA_TO_DEVICE); dma_unmap_sg(dev, src, src_nents, DMA_TO_DEVICE);
dma_unmap_sg(dev, dst, dst_nents, DMA_FROM_DEVICE); if (dst_nents)
dma_unmap_sg(dev, dst, dst_nents, DMA_FROM_DEVICE);
} else { } else {
dma_unmap_sg(dev, src, src_nents, DMA_BIDIRECTIONAL); dma_unmap_sg(dev, src, src_nents, DMA_BIDIRECTIONAL);
} }
@ -961,8 +991,9 @@ static void skcipher_encrypt_done(struct device *jrdev, u32 *desc, u32 err,
* The crypto API expects us to set the IV (req->iv) to the last * The crypto API expects us to set the IV (req->iv) to the last
* ciphertext block. This is used e.g. by the CTS mode. * ciphertext block. This is used e.g. by the CTS mode.
*/ */
scatterwalk_map_and_copy(req->iv, req->dst, req->cryptlen - ivsize, if (ivsize)
ivsize, 0); scatterwalk_map_and_copy(req->iv, req->dst, req->cryptlen -
ivsize, ivsize, 0);
kfree(edesc); kfree(edesc);
@ -1023,11 +1054,12 @@ static void init_aead_job(struct aead_request *req,
init_job_desc_shared(desc, ptr, len, HDR_SHARE_DEFER | HDR_REVERSE); init_job_desc_shared(desc, ptr, len, HDR_SHARE_DEFER | HDR_REVERSE);
if (all_contig) { if (all_contig) {
src_dma = edesc->src_nents ? sg_dma_address(req->src) : 0; src_dma = edesc->mapped_src_nents ? sg_dma_address(req->src) :
0;
in_options = 0; in_options = 0;
} else { } else {
src_dma = edesc->sec4_sg_dma; src_dma = edesc->sec4_sg_dma;
sec4_sg_index += edesc->src_nents; sec4_sg_index += edesc->mapped_src_nents;
in_options = LDST_SGF; in_options = LDST_SGF;
} }
@ -1038,8 +1070,11 @@ static void init_aead_job(struct aead_request *req,
out_options = in_options; out_options = in_options;
if (unlikely(req->src != req->dst)) { if (unlikely(req->src != req->dst)) {
if (edesc->dst_nents == 1) { if (!edesc->mapped_dst_nents) {
dst_dma = 0;
} else if (edesc->mapped_dst_nents == 1) {
dst_dma = sg_dma_address(req->dst); dst_dma = sg_dma_address(req->dst);
out_options = 0;
} else { } else {
dst_dma = edesc->sec4_sg_dma + dst_dma = edesc->sec4_sg_dma +
sec4_sg_index * sec4_sg_index *
@ -1183,9 +1218,9 @@ static void init_skcipher_job(struct skcipher_request *req,
int ivsize = crypto_skcipher_ivsize(skcipher); int ivsize = crypto_skcipher_ivsize(skcipher);
u32 *desc = edesc->hw_desc; u32 *desc = edesc->hw_desc;
u32 *sh_desc; u32 *sh_desc;
u32 out_options = 0; u32 in_options = 0, out_options = 0;
dma_addr_t dst_dma, ptr; dma_addr_t src_dma, dst_dma, ptr;
int len; int len, sec4_sg_index = 0;
#ifdef DEBUG #ifdef DEBUG
print_hex_dump(KERN_ERR, "presciv@"__stringify(__LINE__)": ", print_hex_dump(KERN_ERR, "presciv@"__stringify(__LINE__)": ",
@ -1203,21 +1238,27 @@ static void init_skcipher_job(struct skcipher_request *req,
len = desc_len(sh_desc); len = desc_len(sh_desc);
init_job_desc_shared(desc, ptr, len, HDR_SHARE_DEFER | HDR_REVERSE); init_job_desc_shared(desc, ptr, len, HDR_SHARE_DEFER | HDR_REVERSE);
append_seq_in_ptr(desc, edesc->sec4_sg_dma, req->cryptlen + ivsize, if (ivsize || edesc->mapped_src_nents > 1) {
LDST_SGF); src_dma = edesc->sec4_sg_dma;
sec4_sg_index = edesc->mapped_src_nents + !!ivsize;
in_options = LDST_SGF;
} else {
src_dma = sg_dma_address(req->src);
}
append_seq_in_ptr(desc, src_dma, req->cryptlen + ivsize, in_options);
if (likely(req->src == req->dst)) { if (likely(req->src == req->dst)) {
dst_dma = edesc->sec4_sg_dma + sizeof(struct sec4_sg_entry); dst_dma = src_dma + !!ivsize * sizeof(struct sec4_sg_entry);
out_options = LDST_SGF; out_options = in_options;
} else if (edesc->mapped_dst_nents == 1) {
dst_dma = sg_dma_address(req->dst);
} else { } else {
if (edesc->dst_nents == 1) { dst_dma = edesc->sec4_sg_dma + sec4_sg_index *
dst_dma = sg_dma_address(req->dst); sizeof(struct sec4_sg_entry);
} else { out_options = LDST_SGF;
dst_dma = edesc->sec4_sg_dma + (edesc->src_nents + 1) *
sizeof(struct sec4_sg_entry);
out_options = LDST_SGF;
}
} }
append_seq_out_ptr(desc, dst_dma, req->cryptlen, out_options); append_seq_out_ptr(desc, dst_dma, req->cryptlen, out_options);
} }
@ -1289,12 +1330,19 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
mapped_src_nents = 0; mapped_src_nents = 0;
} }
mapped_dst_nents = dma_map_sg(jrdev, req->dst, dst_nents, /* Cover also the case of null (zero length) output data */
DMA_FROM_DEVICE); if (dst_nents) {
if (unlikely(!mapped_dst_nents)) { mapped_dst_nents = dma_map_sg(jrdev, req->dst,
dev_err(jrdev, "unable to map destination\n"); dst_nents,
dma_unmap_sg(jrdev, req->src, src_nents, DMA_TO_DEVICE); DMA_FROM_DEVICE);
return ERR_PTR(-ENOMEM); if (unlikely(!mapped_dst_nents)) {
dev_err(jrdev, "unable to map destination\n");
dma_unmap_sg(jrdev, req->src, src_nents,
DMA_TO_DEVICE);
return ERR_PTR(-ENOMEM);
}
} else {
mapped_dst_nents = 0;
} }
} }
@ -1313,6 +1361,8 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
edesc->src_nents = src_nents; edesc->src_nents = src_nents;
edesc->dst_nents = dst_nents; edesc->dst_nents = dst_nents;
edesc->mapped_src_nents = mapped_src_nents;
edesc->mapped_dst_nents = mapped_dst_nents;
edesc->sec4_sg = (void *)edesc + sizeof(struct aead_edesc) + edesc->sec4_sg = (void *)edesc + sizeof(struct aead_edesc) +
desc_bytes; desc_bytes;
*all_contig_ptr = !(mapped_src_nents > 1); *all_contig_ptr = !(mapped_src_nents > 1);
@ -1586,7 +1636,7 @@ static struct skcipher_edesc *skcipher_edesc_alloc(struct skcipher_request *req,
GFP_KERNEL : GFP_ATOMIC; GFP_KERNEL : GFP_ATOMIC;
int src_nents, mapped_src_nents, dst_nents = 0, mapped_dst_nents = 0; int src_nents, mapped_src_nents, dst_nents = 0, mapped_dst_nents = 0;
struct skcipher_edesc *edesc; struct skcipher_edesc *edesc;
dma_addr_t iv_dma; dma_addr_t iv_dma = 0;
u8 *iv; u8 *iv;
int ivsize = crypto_skcipher_ivsize(skcipher); int ivsize = crypto_skcipher_ivsize(skcipher);
int dst_sg_idx, sec4_sg_ents, sec4_sg_bytes; int dst_sg_idx, sec4_sg_ents, sec4_sg_bytes;
@ -1621,7 +1671,6 @@ static struct skcipher_edesc *skcipher_edesc_alloc(struct skcipher_request *req,
dev_err(jrdev, "unable to map source\n"); dev_err(jrdev, "unable to map source\n");
return ERR_PTR(-ENOMEM); return ERR_PTR(-ENOMEM);
} }
mapped_dst_nents = dma_map_sg(jrdev, req->dst, dst_nents, mapped_dst_nents = dma_map_sg(jrdev, req->dst, dst_nents,
DMA_FROM_DEVICE); DMA_FROM_DEVICE);
if (unlikely(!mapped_dst_nents)) { if (unlikely(!mapped_dst_nents)) {
@ -1631,7 +1680,10 @@ static struct skcipher_edesc *skcipher_edesc_alloc(struct skcipher_request *req,
} }
} }
sec4_sg_ents = 1 + mapped_src_nents; if (!ivsize && mapped_src_nents == 1)
sec4_sg_ents = 0; // no need for an input hw s/g table
else
sec4_sg_ents = mapped_src_nents + !!ivsize;
dst_sg_idx = sec4_sg_ents; dst_sg_idx = sec4_sg_ents;
sec4_sg_ents += mapped_dst_nents > 1 ? mapped_dst_nents : 0; sec4_sg_ents += mapped_dst_nents > 1 ? mapped_dst_nents : 0;
sec4_sg_bytes = sec4_sg_ents * sizeof(struct sec4_sg_entry); sec4_sg_bytes = sec4_sg_ents * sizeof(struct sec4_sg_entry);
@ -1650,39 +1702,48 @@ static struct skcipher_edesc *skcipher_edesc_alloc(struct skcipher_request *req,
edesc->src_nents = src_nents; edesc->src_nents = src_nents;
edesc->dst_nents = dst_nents; edesc->dst_nents = dst_nents;
edesc->mapped_src_nents = mapped_src_nents;
edesc->mapped_dst_nents = mapped_dst_nents;
edesc->sec4_sg_bytes = sec4_sg_bytes; edesc->sec4_sg_bytes = sec4_sg_bytes;
edesc->sec4_sg = (struct sec4_sg_entry *)((u8 *)edesc->hw_desc + edesc->sec4_sg = (struct sec4_sg_entry *)((u8 *)edesc->hw_desc +
desc_bytes); desc_bytes);
/* Make sure IV is located in a DMAable area */ /* Make sure IV is located in a DMAable area */
iv = (u8 *)edesc->hw_desc + desc_bytes + sec4_sg_bytes; if (ivsize) {
memcpy(iv, req->iv, ivsize); iv = (u8 *)edesc->hw_desc + desc_bytes + sec4_sg_bytes;
memcpy(iv, req->iv, ivsize);
iv_dma = dma_map_single(jrdev, iv, ivsize, DMA_TO_DEVICE); iv_dma = dma_map_single(jrdev, iv, ivsize, DMA_TO_DEVICE);
if (dma_mapping_error(jrdev, iv_dma)) { if (dma_mapping_error(jrdev, iv_dma)) {
dev_err(jrdev, "unable to map IV\n"); dev_err(jrdev, "unable to map IV\n");
caam_unmap(jrdev, req->src, req->dst, src_nents, dst_nents, 0, caam_unmap(jrdev, req->src, req->dst, src_nents,
0, 0, 0); dst_nents, 0, 0, 0, 0);
kfree(edesc); kfree(edesc);
return ERR_PTR(-ENOMEM); return ERR_PTR(-ENOMEM);
}
dma_to_sec4_sg_one(edesc->sec4_sg, iv_dma, ivsize, 0);
} }
if (dst_sg_idx)
dma_to_sec4_sg_one(edesc->sec4_sg, iv_dma, ivsize, 0); sg_to_sec4_sg_last(req->src, mapped_src_nents, edesc->sec4_sg +
sg_to_sec4_sg_last(req->src, mapped_src_nents, edesc->sec4_sg + 1, 0); !!ivsize, 0);
if (mapped_dst_nents > 1) { if (mapped_dst_nents > 1) {
sg_to_sec4_sg_last(req->dst, mapped_dst_nents, sg_to_sec4_sg_last(req->dst, mapped_dst_nents,
edesc->sec4_sg + dst_sg_idx, 0); edesc->sec4_sg + dst_sg_idx, 0);
} }
edesc->sec4_sg_dma = dma_map_single(jrdev, edesc->sec4_sg, if (sec4_sg_bytes) {
sec4_sg_bytes, DMA_TO_DEVICE); edesc->sec4_sg_dma = dma_map_single(jrdev, edesc->sec4_sg,
if (dma_mapping_error(jrdev, edesc->sec4_sg_dma)) { sec4_sg_bytes,
dev_err(jrdev, "unable to map S/G table\n"); DMA_TO_DEVICE);
caam_unmap(jrdev, req->src, req->dst, src_nents, dst_nents, if (dma_mapping_error(jrdev, edesc->sec4_sg_dma)) {
iv_dma, ivsize, 0, 0); dev_err(jrdev, "unable to map S/G table\n");
kfree(edesc); caam_unmap(jrdev, req->src, req->dst, src_nents,
return ERR_PTR(-ENOMEM); dst_nents, iv_dma, ivsize, 0, 0);
kfree(edesc);
return ERR_PTR(-ENOMEM);
}
} }
edesc->iv_dma = iv_dma; edesc->iv_dma = iv_dma;
@ -1749,8 +1810,9 @@ static int skcipher_decrypt(struct skcipher_request *req)
* The crypto API expects us to set the IV (req->iv) to the last * The crypto API expects us to set the IV (req->iv) to the last
* ciphertext block. * ciphertext block.
*/ */
scatterwalk_map_and_copy(req->iv, req->src, req->cryptlen - ivsize, if (ivsize)
ivsize, 0); scatterwalk_map_and_copy(req->iv, req->src, req->cryptlen -
ivsize, ivsize, 0);
/* Create and submit job descriptor*/ /* Create and submit job descriptor*/
init_skcipher_job(req, edesc, false); init_skcipher_job(req, edesc, false);
@ -1796,7 +1858,7 @@ static struct caam_skcipher_alg driver_algs[] = {
.cra_driver_name = "cbc-3des-caam", .cra_driver_name = "cbc-3des-caam",
.cra_blocksize = DES3_EDE_BLOCK_SIZE, .cra_blocksize = DES3_EDE_BLOCK_SIZE,
}, },
.setkey = skcipher_setkey, .setkey = des_skcipher_setkey,
.encrypt = skcipher_encrypt, .encrypt = skcipher_encrypt,
.decrypt = skcipher_decrypt, .decrypt = skcipher_decrypt,
.min_keysize = DES3_EDE_KEY_SIZE, .min_keysize = DES3_EDE_KEY_SIZE,
@ -1812,7 +1874,7 @@ static struct caam_skcipher_alg driver_algs[] = {
.cra_driver_name = "cbc-des-caam", .cra_driver_name = "cbc-des-caam",
.cra_blocksize = DES_BLOCK_SIZE, .cra_blocksize = DES_BLOCK_SIZE,
}, },
.setkey = skcipher_setkey, .setkey = des_skcipher_setkey,
.encrypt = skcipher_encrypt, .encrypt = skcipher_encrypt,
.decrypt = skcipher_decrypt, .decrypt = skcipher_decrypt,
.min_keysize = DES_KEY_SIZE, .min_keysize = DES_KEY_SIZE,
@ -1878,6 +1940,66 @@ static struct caam_skcipher_alg driver_algs[] = {
}, },
.caam.class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_XTS, .caam.class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_XTS,
}, },
{
.skcipher = {
.base = {
.cra_name = "ecb(des)",
.cra_driver_name = "ecb-des-caam",
.cra_blocksize = DES_BLOCK_SIZE,
},
.setkey = des_skcipher_setkey,
.encrypt = skcipher_encrypt,
.decrypt = skcipher_decrypt,
.min_keysize = DES_KEY_SIZE,
.max_keysize = DES_KEY_SIZE,
},
.caam.class1_alg_type = OP_ALG_ALGSEL_DES | OP_ALG_AAI_ECB,
},
{
.skcipher = {
.base = {
.cra_name = "ecb(aes)",
.cra_driver_name = "ecb-aes-caam",
.cra_blocksize = AES_BLOCK_SIZE,
},
.setkey = skcipher_setkey,
.encrypt = skcipher_encrypt,
.decrypt = skcipher_decrypt,
.min_keysize = AES_MIN_KEY_SIZE,
.max_keysize = AES_MAX_KEY_SIZE,
},
.caam.class1_alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_ECB,
},
{
.skcipher = {
.base = {
.cra_name = "ecb(des3_ede)",
.cra_driver_name = "ecb-des3-caam",
.cra_blocksize = DES3_EDE_BLOCK_SIZE,
},
.setkey = des_skcipher_setkey,
.encrypt = skcipher_encrypt,
.decrypt = skcipher_decrypt,
.min_keysize = DES3_EDE_KEY_SIZE,
.max_keysize = DES3_EDE_KEY_SIZE,
},
.caam.class1_alg_type = OP_ALG_ALGSEL_3DES | OP_ALG_AAI_ECB,
},
{
.skcipher = {
.base = {
.cra_name = "ecb(arc4)",
.cra_driver_name = "ecb-arc4-caam",
.cra_blocksize = ARC4_BLOCK_SIZE,
},
.setkey = skcipher_setkey,
.encrypt = skcipher_encrypt,
.decrypt = skcipher_decrypt,
.min_keysize = ARC4_MIN_KEY_SIZE,
.max_keysize = ARC4_MAX_KEY_SIZE,
},
.caam.class1_alg_type = OP_ALG_ALGSEL_ARC4 | OP_ALG_AAI_ECB,
},
}; };
static struct caam_aead_alg driver_aeads[] = { static struct caam_aead_alg driver_aeads[] = {
@ -3337,6 +3459,7 @@ static int __init caam_algapi_init(void)
struct caam_drv_private *priv; struct caam_drv_private *priv;
int i = 0, err = 0; int i = 0, err = 0;
u32 aes_vid, aes_inst, des_inst, md_vid, md_inst, ccha_inst, ptha_inst; u32 aes_vid, aes_inst, des_inst, md_vid, md_inst, ccha_inst, ptha_inst;
u32 arc4_inst;
unsigned int md_limit = SHA512_DIGEST_SIZE; unsigned int md_limit = SHA512_DIGEST_SIZE;
bool registered = false; bool registered = false;
@ -3381,6 +3504,8 @@ static int __init caam_algapi_init(void)
CHA_ID_LS_DES_SHIFT; CHA_ID_LS_DES_SHIFT;
aes_inst = cha_inst & CHA_ID_LS_AES_MASK; aes_inst = cha_inst & CHA_ID_LS_AES_MASK;
md_inst = (cha_inst & CHA_ID_LS_MD_MASK) >> CHA_ID_LS_MD_SHIFT; md_inst = (cha_inst & CHA_ID_LS_MD_MASK) >> CHA_ID_LS_MD_SHIFT;
arc4_inst = (cha_inst & CHA_ID_LS_ARC4_MASK) >>
CHA_ID_LS_ARC4_SHIFT;
ccha_inst = 0; ccha_inst = 0;
ptha_inst = 0; ptha_inst = 0;
} else { } else {
@ -3397,6 +3522,7 @@ static int __init caam_algapi_init(void)
md_inst = mdha & CHA_VER_NUM_MASK; md_inst = mdha & CHA_VER_NUM_MASK;
ccha_inst = rd_reg32(&priv->ctrl->vreg.ccha) & CHA_VER_NUM_MASK; ccha_inst = rd_reg32(&priv->ctrl->vreg.ccha) & CHA_VER_NUM_MASK;
ptha_inst = rd_reg32(&priv->ctrl->vreg.ptha) & CHA_VER_NUM_MASK; ptha_inst = rd_reg32(&priv->ctrl->vreg.ptha) & CHA_VER_NUM_MASK;
arc4_inst = rd_reg32(&priv->ctrl->vreg.afha) & CHA_VER_NUM_MASK;
} }
/* If MD is present, limit digest size based on LP256 */ /* If MD is present, limit digest size based on LP256 */
@ -3417,6 +3543,10 @@ static int __init caam_algapi_init(void)
if (!aes_inst && (alg_sel == OP_ALG_ALGSEL_AES)) if (!aes_inst && (alg_sel == OP_ALG_ALGSEL_AES))
continue; continue;
/* Skip ARC4 algorithms if not supported by device */
if (!arc4_inst && alg_sel == OP_ALG_ALGSEL_ARC4)
continue;
/* /*
* Check support for AES modes not available * Check support for AES modes not available
* on LP devices. * on LP devices.

18
drivers/crypto/caam/caamalg_desc.c
View File

@ -2,7 +2,7 @@
/* /*
* Shared descriptors for aead, skcipher algorithms * Shared descriptors for aead, skcipher algorithms
* *
* Copyright 2016-2018 NXP * Copyright 2016-2019 NXP
*/ */
#include "compat.h" #include "compat.h"
@ -1396,9 +1396,11 @@ void cnstr_shdsc_skcipher_encap(u32 * const desc, struct alginfo *cdata,
set_jump_tgt_here(desc, key_jump_cmd); set_jump_tgt_here(desc, key_jump_cmd);
/* Load iv */ /* Load IV, if there is one */
append_seq_load(desc, ivsize, LDST_SRCDST_BYTE_CONTEXT | if (ivsize)
LDST_CLASS_1_CCB | (ctx1_iv_off << LDST_OFFSET_SHIFT)); append_seq_load(desc, ivsize, LDST_SRCDST_BYTE_CONTEXT |
LDST_CLASS_1_CCB | (ctx1_iv_off <<
LDST_OFFSET_SHIFT));
/* Load counter into CONTEXT1 reg */ /* Load counter into CONTEXT1 reg */
if (is_rfc3686) if (is_rfc3686)
@ -1462,9 +1464,11 @@ void cnstr_shdsc_skcipher_decap(u32 * const desc, struct alginfo *cdata,
set_jump_tgt_here(desc, key_jump_cmd); set_jump_tgt_here(desc, key_jump_cmd);
/* load IV */ /* Load IV, if there is one */
append_seq_load(desc, ivsize, LDST_SRCDST_BYTE_CONTEXT | if (ivsize)
LDST_CLASS_1_CCB | (ctx1_iv_off << LDST_OFFSET_SHIFT)); append_seq_load(desc, ivsize, LDST_SRCDST_BYTE_CONTEXT |
LDST_CLASS_1_CCB | (ctx1_iv_off <<
LDST_OFFSET_SHIFT));
/* Load counter into CONTEXT1 reg */ /* Load counter into CONTEXT1 reg */
if (is_rfc3686) if (is_rfc3686)

29
drivers/crypto/caam/caamalg_qi.c
View File

@ -782,7 +782,7 @@ static struct caam_drv_ctx *get_drv_ctx(struct caam_ctx *ctx,
cpu = smp_processor_id(); cpu = smp_processor_id();
drv_ctx = caam_drv_ctx_init(ctx->qidev, &cpu, desc); drv_ctx = caam_drv_ctx_init(ctx->qidev, &cpu, desc);
if (likely(!IS_ERR_OR_NULL(drv_ctx))) if (!IS_ERR_OR_NULL(drv_ctx))
drv_ctx->op_type = type; drv_ctx->op_type = type;
ctx->drv_ctx[type] = drv_ctx; ctx->drv_ctx[type] = drv_ctx;
@ -802,7 +802,8 @@ static void caam_unmap(struct device *dev, struct scatterlist *src,
if (dst != src) { if (dst != src) {
if (src_nents) if (src_nents)
dma_unmap_sg(dev, src, src_nents, DMA_TO_DEVICE); dma_unmap_sg(dev, src, src_nents, DMA_TO_DEVICE);
dma_unmap_sg(dev, dst, dst_nents, DMA_FROM_DEVICE); if (dst_nents)
dma_unmap_sg(dev, dst, dst_nents, DMA_FROM_DEVICE);
} else { } else {
dma_unmap_sg(dev, src, src_nents, DMA_BIDIRECTIONAL); dma_unmap_sg(dev, src, src_nents, DMA_BIDIRECTIONAL);
} }
@ -892,7 +893,7 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
struct caam_drv_ctx *drv_ctx; struct caam_drv_ctx *drv_ctx;
drv_ctx = get_drv_ctx(ctx, encrypt ? ENCRYPT : DECRYPT); drv_ctx = get_drv_ctx(ctx, encrypt ? ENCRYPT : DECRYPT);
if (unlikely(IS_ERR_OR_NULL(drv_ctx))) if (IS_ERR_OR_NULL(drv_ctx))
return (struct aead_edesc *)drv_ctx; return (struct aead_edesc *)drv_ctx;
/* allocate space for base edesc and hw desc commands, link tables */ /* allocate space for base edesc and hw desc commands, link tables */
@ -955,13 +956,19 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
mapped_src_nents = 0; mapped_src_nents = 0;
} }
mapped_dst_nents = dma_map_sg(qidev, req->dst, dst_nents, if (dst_nents) {
DMA_FROM_DEVICE); mapped_dst_nents = dma_map_sg(qidev, req->dst,
if (unlikely(!mapped_dst_nents)) { dst_nents,
dev_err(qidev, "unable to map destination\n"); DMA_FROM_DEVICE);
dma_unmap_sg(qidev, req->src, src_nents, DMA_TO_DEVICE); if (unlikely(!mapped_dst_nents)) {
qi_cache_free(edesc); dev_err(qidev, "unable to map destination\n");
return ERR_PTR(-ENOMEM); dma_unmap_sg(qidev, req->src, src_nents,
DMA_TO_DEVICE);
qi_cache_free(edesc);
return ERR_PTR(-ENOMEM);
}
} else {
mapped_dst_nents = 0;
} }
} }
@ -1184,7 +1191,7 @@ static struct skcipher_edesc *skcipher_edesc_alloc(struct skcipher_request *req,
struct caam_drv_ctx *drv_ctx; struct caam_drv_ctx *drv_ctx;
drv_ctx = get_drv_ctx(ctx, encrypt ? ENCRYPT : DECRYPT); drv_ctx = get_drv_ctx(ctx, encrypt ? ENCRYPT : DECRYPT);
if (unlikely(IS_ERR_OR_NULL(drv_ctx))) if (IS_ERR_OR_NULL(drv_ctx))
return (struct skcipher_edesc *)drv_ctx; return (struct skcipher_edesc *)drv_ctx;
src_nents = sg_nents_for_len(req->src, req->cryptlen); src_nents = sg_nents_for_len(req->src, req->cryptlen);

85
drivers/crypto/caam/caamalg_qi2.c
View File

@ -25,13 +25,6 @@
#define CAAM_MAX_KEY_SIZE (AES_MAX_KEY_SIZE + CTR_RFC3686_NONCE_SIZE + \ #define CAAM_MAX_KEY_SIZE (AES_MAX_KEY_SIZE + CTR_RFC3686_NONCE_SIZE + \
SHA512_DIGEST_SIZE * 2) SHA512_DIGEST_SIZE * 2)
#if !IS_ENABLED(CONFIG_CRYPTO_DEV_FSL_CAAM)
bool caam_little_end;
EXPORT_SYMBOL(caam_little_end);
bool caam_imx;
EXPORT_SYMBOL(caam_imx);
#endif
/* /*
* This is a a cache of buffers, from which the users of CAAM QI driver * This is a a cache of buffers, from which the users of CAAM QI driver
* can allocate short buffers. It's speedier than doing kmalloc on the hotpath. * can allocate short buffers. It's speedier than doing kmalloc on the hotpath.
@ -151,7 +144,8 @@ static void caam_unmap(struct device *dev, struct scatterlist *src,
if (dst != src) { if (dst != src) {
if (src_nents) if (src_nents)
dma_unmap_sg(dev, src, src_nents, DMA_TO_DEVICE); dma_unmap_sg(dev, src, src_nents, DMA_TO_DEVICE);
dma_unmap_sg(dev, dst, dst_nents, DMA_FROM_DEVICE); if (dst_nents)
dma_unmap_sg(dev, dst, dst_nents, DMA_FROM_DEVICE);
} else { } else {
dma_unmap_sg(dev, src, src_nents, DMA_BIDIRECTIONAL); dma_unmap_sg(dev, src, src_nents, DMA_BIDIRECTIONAL);
} }
@ -392,13 +386,18 @@ static struct aead_edesc *aead_edesc_alloc(struct aead_request *req,
mapped_src_nents = 0; mapped_src_nents = 0;
} }
mapped_dst_nents = dma_map_sg(dev, req->dst, dst_nents, if (dst_nents) {
DMA_FROM_DEVICE); mapped_dst_nents = dma_map_sg(dev, req->dst, dst_nents,
if (unlikely(!mapped_dst_nents)) { DMA_FROM_DEVICE);
dev_err(dev, "unable to map destination\n"); if (unlikely(!mapped_dst_nents)) {
dma_unmap_sg(dev, req->src, src_nents, DMA_TO_DEVICE); dev_err(dev, "unable to map destination\n");
qi_cache_free(edesc); dma_unmap_sg(dev, req->src, src_nents,
return ERR_PTR(-ENOMEM); DMA_TO_DEVICE);
qi_cache_free(edesc);
return ERR_PTR(-ENOMEM);
}
} else {
mapped_dst_nents = 0;
} }
} else { } else {
src_nents = sg_nents_for_len(req->src, req->assoclen + src_nents = sg_nents_for_len(req->src, req->assoclen +
@ -4503,7 +4502,8 @@ static int __cold dpaa2_dpseci_dpio_setup(struct dpaa2_caam_priv *priv)
nctx->cb = dpaa2_caam_fqdan_cb; nctx->cb = dpaa2_caam_fqdan_cb;
/* Register notification callbacks */ /* Register notification callbacks */
err = dpaa2_io_service_register(NULL, nctx); ppriv->dpio = dpaa2_io_service_select(cpu);
err = dpaa2_io_service_register(ppriv->dpio, nctx, dev);
if (unlikely(err)) { if (unlikely(err)) {
dev_dbg(dev, "No affine DPIO for cpu %d\n", cpu); dev_dbg(dev, "No affine DPIO for cpu %d\n", cpu);
nctx->cb = NULL; nctx->cb = NULL;
@ -4536,7 +4536,7 @@ err:
ppriv = per_cpu_ptr(priv->ppriv, cpu); ppriv = per_cpu_ptr(priv->ppriv, cpu);
if (!ppriv->nctx.cb) if (!ppriv->nctx.cb)
break; break;
dpaa2_io_service_deregister(NULL, &ppriv->nctx); dpaa2_io_service_deregister(ppriv->dpio, &ppriv->nctx, dev);
} }
for_each_online_cpu(cpu) { for_each_online_cpu(cpu) {
@ -4556,7 +4556,8 @@ static void __cold dpaa2_dpseci_dpio_free(struct dpaa2_caam_priv *priv)
for_each_online_cpu(cpu) { for_each_online_cpu(cpu) {
ppriv = per_cpu_ptr(priv->ppriv, cpu); ppriv = per_cpu_ptr(priv->ppriv, cpu);
dpaa2_io_service_deregister(NULL, &ppriv->nctx); dpaa2_io_service_deregister(ppriv->dpio, &ppriv->nctx,
priv->dev);
dpaa2_io_store_destroy(ppriv->store); dpaa2_io_store_destroy(ppriv->store);
if (++i == priv->num_pairs) if (++i == priv->num_pairs)
@ -4654,7 +4655,7 @@ static int dpaa2_caam_pull_fq(struct dpaa2_caam_priv_per_cpu *ppriv)
/* Retry while portal is busy */ /* Retry while portal is busy */
do { do {
err = dpaa2_io_service_pull_fq(NULL, ppriv->rsp_fqid, err = dpaa2_io_service_pull_fq(ppriv->dpio, ppriv->rsp_fqid,
ppriv->store); ppriv->store);
} while (err == -EBUSY); } while (err == -EBUSY);
@ -4722,7 +4723,7 @@ static int dpaa2_dpseci_poll(struct napi_struct *napi, int budget)
if (cleaned < budget) { if (cleaned < budget) {
napi_complete_done(napi, cleaned); napi_complete_done(napi, cleaned);
err = dpaa2_io_service_rearm(NULL, &ppriv->nctx); err = dpaa2_io_service_rearm(ppriv->dpio, &ppriv->nctx);
if (unlikely(err)) if (unlikely(err))
dev_err(priv->dev, "Notification rearm failed: %d\n", dev_err(priv->dev, "Notification rearm failed: %d\n",
err); err);
@ -4863,21 +4864,31 @@ static int __cold dpaa2_dpseci_setup(struct fsl_mc_device *ls_dev)
i = 0; i = 0;
for_each_online_cpu(cpu) { for_each_online_cpu(cpu) {
dev_dbg(dev, "pair %d: rx queue %d, tx queue %d\n", i, u8 j;
priv->rx_queue_attr[i].fqid,
priv->tx_queue_attr[i].fqid); j = i % priv->num_pairs;
ppriv = per_cpu_ptr(priv->ppriv, cpu); ppriv = per_cpu_ptr(priv->ppriv, cpu);
ppriv->req_fqid = priv->tx_queue_attr[i].fqid; ppriv->req_fqid = priv->tx_queue_attr[j].fqid;
ppriv->rsp_fqid = priv->rx_queue_attr[i].fqid;
ppriv->prio = i; /*
* Allow all cores to enqueue, while only some of them
* will take part in dequeuing.
*/
if (++i > priv->num_pairs)
continue;
ppriv->rsp_fqid = priv->rx_queue_attr[j].fqid;
ppriv->prio = j;
dev_dbg(dev, "pair %d: rx queue %d, tx queue %d\n", j,
priv->rx_queue_attr[j].fqid,
priv->tx_queue_attr[j].fqid);
ppriv->net_dev.dev = *dev; ppriv->net_dev.dev = *dev;
INIT_LIST_HEAD(&ppriv->net_dev.napi_list); INIT_LIST_HEAD(&ppriv->net_dev.napi_list);
netif_napi_add(&ppriv->net_dev, &ppriv->napi, dpaa2_dpseci_poll, netif_napi_add(&ppriv->net_dev, &ppriv->napi, dpaa2_dpseci_poll,
DPAA2_CAAM_NAPI_WEIGHT); DPAA2_CAAM_NAPI_WEIGHT);
if (++i == priv->num_pairs)
break;
} }
return 0; return 0;
@ -5229,7 +5240,8 @@ int dpaa2_caam_enqueue(struct device *dev, struct caam_request *req)
{ {
struct dpaa2_fd fd; struct dpaa2_fd fd;
struct dpaa2_caam_priv *priv = dev_get_drvdata(dev); struct dpaa2_caam_priv *priv = dev_get_drvdata(dev);
int err = 0, i, id; struct dpaa2_caam_priv_per_cpu *ppriv;
int err = 0, i;
if (IS_ERR(req)) if (IS_ERR(req))
return PTR_ERR(req); return PTR_ERR(req);
@ -5259,23 +5271,18 @@ int dpaa2_caam_enqueue(struct device *dev, struct caam_request *req)
dpaa2_fd_set_len(&fd, dpaa2_fl_get_len(&req->fd_flt[1])); dpaa2_fd_set_len(&fd, dpaa2_fl_get_len(&req->fd_flt[1]));
dpaa2_fd_set_flc(&fd, req->flc_dma); dpaa2_fd_set_flc(&fd, req->flc_dma);
/* ppriv = this_cpu_ptr(priv->ppriv);
* There is no guarantee that preemption is disabled here,
* thus take action.
*/
preempt_disable();
id = smp_processor_id() % priv->dpseci_attr.num_tx_queues;
for (i = 0; i < (priv->dpseci_attr.num_tx_queues << 1); i++) { for (i = 0; i < (priv->dpseci_attr.num_tx_queues << 1); i++) {
err = dpaa2_io_service_enqueue_fq(NULL, err = dpaa2_io_service_enqueue_fq(ppriv->dpio, ppriv->req_fqid,
priv->tx_queue_attr[id].fqid,
&fd); &fd);
if (err != -EBUSY) if (err != -EBUSY)
break; break;
cpu_relax();
} }
preempt_enable();
if (unlikely(err)) { if (unlikely(err)) {
dev_err(dev, "Error enqueuing frame: %d\n", err); dev_err_ratelimited(dev, "Error enqueuing frame: %d\n", err);
goto err_out; goto err_out;
} }

2
drivers/crypto/caam/caamalg_qi2.h
View File

@ -76,6 +76,7 @@ struct dpaa2_caam_priv {
* @nctx: notification context of response FQ * @nctx: notification context of response FQ
* @store: where dequeued frames are stored * @store: where dequeued frames are stored
* @priv: backpointer to dpaa2_caam_priv * @priv: backpointer to dpaa2_caam_priv
* @dpio: portal used for data path operations
*/ */
struct dpaa2_caam_priv_per_cpu { struct dpaa2_caam_priv_per_cpu {
struct napi_struct napi; struct napi_struct napi;
@ -86,6 +87,7 @@ struct dpaa2_caam_priv_per_cpu {
struct dpaa2_io_notification_ctx nctx; struct dpaa2_io_notification_ctx nctx;
struct dpaa2_io_store *store; struct dpaa2_io_store *store;
struct dpaa2_caam_priv *priv; struct dpaa2_caam_priv *priv;
struct dpaa2_io *dpio;
}; };
/* /*

429
drivers/crypto/caam/caamhash.c
View File

@ -3,7 +3,7 @@
* caam - Freescale FSL CAAM support for ahash functions of crypto API * caam - Freescale FSL CAAM support for ahash functions of crypto API
* *
* Copyright 2011 Freescale Semiconductor, Inc. * Copyright 2011 Freescale Semiconductor, Inc.
* Copyright 2018 NXP * Copyright 2018-2019 NXP
* *
* Based on caamalg.c crypto API driver. * Based on caamalg.c crypto API driver.
* *
@ -98,13 +98,14 @@ struct caam_hash_ctx {
u32 sh_desc_update_first[DESC_HASH_MAX_USED_LEN] ____cacheline_aligned; u32 sh_desc_update_first[DESC_HASH_MAX_USED_LEN] ____cacheline_aligned;
u32 sh_desc_fin[DESC_HASH_MAX_USED_LEN] ____cacheline_aligned; u32 sh_desc_fin[DESC_HASH_MAX_USED_LEN] ____cacheline_aligned;
u32 sh_desc_digest[DESC_HASH_MAX_USED_LEN] ____cacheline_aligned; u32 sh_desc_digest[DESC_HASH_MAX_USED_LEN] ____cacheline_aligned;
u8 key[CAAM_MAX_HASH_KEY_SIZE] ____cacheline_aligned;
dma_addr_t sh_desc_update_dma ____cacheline_aligned; dma_addr_t sh_desc_update_dma ____cacheline_aligned;
dma_addr_t sh_desc_update_first_dma; dma_addr_t sh_desc_update_first_dma;
dma_addr_t sh_desc_fin_dma; dma_addr_t sh_desc_fin_dma;
dma_addr_t sh_desc_digest_dma; dma_addr_t sh_desc_digest_dma;
dma_addr_t key_dma;
enum dma_data_direction dir; enum dma_data_direction dir;
struct device *jrdev; struct device *jrdev;
u8 key[CAAM_MAX_HASH_KEY_SIZE];
int ctx_len; int ctx_len;
struct alginfo adata; struct alginfo adata;
}; };
@ -113,6 +114,7 @@ struct caam_hash_ctx {
struct caam_hash_state { struct caam_hash_state {
dma_addr_t buf_dma; dma_addr_t buf_dma;
dma_addr_t ctx_dma; dma_addr_t ctx_dma;
int ctx_dma_len;
u8 buf_0[CAAM_MAX_HASH_BLOCK_SIZE] ____cacheline_aligned; u8 buf_0[CAAM_MAX_HASH_BLOCK_SIZE] ____cacheline_aligned;
int buflen_0; int buflen_0;
u8 buf_1[CAAM_MAX_HASH_BLOCK_SIZE] ____cacheline_aligned; u8 buf_1[CAAM_MAX_HASH_BLOCK_SIZE] ____cacheline_aligned;
@ -158,6 +160,11 @@ static inline int *alt_buflen(struct caam_hash_state *state)
return state->current_buf ? &state->buflen_0 : &state->buflen_1; return state->current_buf ? &state->buflen_0 : &state->buflen_1;
} }
static inline bool is_cmac_aes(u32 algtype)
{
return (algtype & (OP_ALG_ALGSEL_MASK | OP_ALG_AAI_MASK)) ==
(OP_ALG_ALGSEL_AES | OP_ALG_AAI_CMAC);
}
/* Common job descriptor seq in/out ptr routines */ /* Common job descriptor seq in/out ptr routines */
/* Map state->caam_ctx, and append seq_out_ptr command that points to it */ /* Map state->caam_ctx, and append seq_out_ptr command that points to it */
@ -165,6 +172,7 @@ static inline int map_seq_out_ptr_ctx(u32 *desc, struct device *jrdev,
struct caam_hash_state *state, struct caam_hash_state *state,
int ctx_len) int ctx_len)
{ {
state->ctx_dma_len = ctx_len;
state->ctx_dma = dma_map_single(jrdev, state->caam_ctx, state->ctx_dma = dma_map_single(jrdev, state->caam_ctx,
ctx_len, DMA_FROM_DEVICE); ctx_len, DMA_FROM_DEVICE);
if (dma_mapping_error(jrdev, state->ctx_dma)) { if (dma_mapping_error(jrdev, state->ctx_dma)) {
@ -178,18 +186,6 @@ static inline int map_seq_out_ptr_ctx(u32 *desc, struct device *jrdev,
return 0; return 0;
} }
/* Map req->result, and append seq_out_ptr command that points to it */
static inline dma_addr_t map_seq_out_ptr_result(u32 *desc, struct device *jrdev,
u8 *result, int digestsize)
{
dma_addr_t dst_dma;
dst_dma = dma_map_single(jrdev, result, digestsize, DMA_FROM_DEVICE);
append_seq_out_ptr(desc, dst_dma, digestsize, 0);
return dst_dma;
}
/* Map current buffer in state (if length > 0) and put it in link table */ /* Map current buffer in state (if length > 0) and put it in link table */
static inline int buf_map_to_sec4_sg(struct device *jrdev, static inline int buf_map_to_sec4_sg(struct device *jrdev,
struct sec4_sg_entry *sec4_sg, struct sec4_sg_entry *sec4_sg,
@ -218,6 +214,7 @@ static inline int ctx_map_to_sec4_sg(struct device *jrdev,
struct caam_hash_state *state, int ctx_len, struct caam_hash_state *state, int ctx_len,
struct sec4_sg_entry *sec4_sg, u32 flag) struct sec4_sg_entry *sec4_sg, u32 flag)
{ {
state->ctx_dma_len = ctx_len;
state->ctx_dma = dma_map_single(jrdev, state->caam_ctx, ctx_len, flag); state->ctx_dma = dma_map_single(jrdev, state->caam_ctx, ctx_len, flag);
if (dma_mapping_error(jrdev, state->ctx_dma)) { if (dma_mapping_error(jrdev, state->ctx_dma)) {
dev_err(jrdev, "unable to map ctx\n"); dev_err(jrdev, "unable to map ctx\n");
@ -292,14 +289,119 @@ static int ahash_set_sh_desc(struct crypto_ahash *ahash)
return 0; return 0;
} }
static int axcbc_set_sh_desc(struct crypto_ahash *ahash)
{
struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
int digestsize = crypto_ahash_digestsize(ahash);
struct device *jrdev = ctx->jrdev;
u32 *desc;
/* key is loaded from memory for UPDATE and FINALIZE states */
ctx->adata.key_dma = ctx->key_dma;
/* shared descriptor for ahash_update */
desc = ctx->sh_desc_update;
cnstr_shdsc_sk_hash(desc, &ctx->adata, OP_ALG_AS_UPDATE,
ctx->ctx_len, ctx->ctx_len, 0);
dma_sync_single_for_device(jrdev, ctx->sh_desc_update_dma,
desc_bytes(desc), ctx->dir);
print_hex_dump_debug("axcbc update shdesc@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc),
1);
/* shared descriptor for ahash_{final,finup} */
desc = ctx->sh_desc_fin;
cnstr_shdsc_sk_hash(desc, &ctx->adata, OP_ALG_AS_FINALIZE,
digestsize, ctx->ctx_len, 0);
dma_sync_single_for_device(jrdev, ctx->sh_desc_fin_dma,
desc_bytes(desc), ctx->dir);
print_hex_dump_debug("axcbc finup shdesc@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc),
1);
/* key is immediate data for INIT and INITFINAL states */
ctx->adata.key_virt = ctx->key;
/* shared descriptor for first invocation of ahash_update */
desc = ctx->sh_desc_update_first;
cnstr_shdsc_sk_hash(desc, &ctx->adata, OP_ALG_AS_INIT, ctx->ctx_len,
ctx->ctx_len, ctx->key_dma);
dma_sync_single_for_device(jrdev, ctx->sh_desc_update_first_dma,
desc_bytes(desc), ctx->dir);
print_hex_dump_debug("axcbc update first shdesc@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc),
1);
/* shared descriptor for ahash_digest */
desc = ctx->sh_desc_digest;
cnstr_shdsc_sk_hash(desc, &ctx->adata, OP_ALG_AS_INITFINAL,
digestsize, ctx->ctx_len, 0);
dma_sync_single_for_device(jrdev, ctx->sh_desc_digest_dma,
desc_bytes(desc), ctx->dir);
print_hex_dump_debug("axcbc digest shdesc@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc),
1);
return 0;
}
static int acmac_set_sh_desc(struct crypto_ahash *ahash)
{
struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
int digestsize = crypto_ahash_digestsize(ahash);
struct device *jrdev = ctx->jrdev;
u32 *desc;
/* shared descriptor for ahash_update */
desc = ctx->sh_desc_update;
cnstr_shdsc_sk_hash(desc, &ctx->adata, OP_ALG_AS_UPDATE,
ctx->ctx_len, ctx->ctx_len, 0);
dma_sync_single_for_device(jrdev, ctx->sh_desc_update_dma,
desc_bytes(desc), ctx->dir);
print_hex_dump_debug("acmac update shdesc@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, desc,
desc_bytes(desc), 1);
/* shared descriptor for ahash_{final,finup} */
desc = ctx->sh_desc_fin;
cnstr_shdsc_sk_hash(desc, &ctx->adata, OP_ALG_AS_FINALIZE,
digestsize, ctx->ctx_len, 0);
dma_sync_single_for_device(jrdev, ctx->sh_desc_fin_dma,
desc_bytes(desc), ctx->dir);
print_hex_dump_debug("acmac finup shdesc@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, desc,
desc_bytes(desc), 1);
/* shared descriptor for first invocation of ahash_update */
desc = ctx->sh_desc_update_first;
cnstr_shdsc_sk_hash(desc, &ctx->adata, OP_ALG_AS_INIT, ctx->ctx_len,
ctx->ctx_len, 0);
dma_sync_single_for_device(jrdev, ctx->sh_desc_update_first_dma,
desc_bytes(desc), ctx->dir);
print_hex_dump_debug("acmac update first shdesc@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, desc,
desc_bytes(desc), 1);
/* shared descriptor for ahash_digest */
desc = ctx->sh_desc_digest;
cnstr_shdsc_sk_hash(desc, &ctx->adata, OP_ALG_AS_INITFINAL,
digestsize, ctx->ctx_len, 0);
dma_sync_single_for_device(jrdev, ctx->sh_desc_digest_dma,
desc_bytes(desc), ctx->dir);
print_hex_dump_debug("acmac digest shdesc@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, desc,
desc_bytes(desc), 1);
return 0;
}
/* Digest hash size if it is too large */ /* Digest hash size if it is too large */
static int hash_digest_key(struct caam_hash_ctx *ctx, const u8 *key_in, static int hash_digest_key(struct caam_hash_ctx *ctx, u32 *keylen, u8 *key,
u32 *keylen, u8 *key_out, u32 digestsize) u32 digestsize)
{ {
struct device *jrdev = ctx->jrdev; struct device *jrdev = ctx->jrdev;
u32 *desc; u32 *desc;
struct split_key_result result; struct split_key_result result;
dma_addr_t src_dma, dst_dma; dma_addr_t key_dma;
int ret; int ret;
desc = kmalloc(CAAM_CMD_SZ * 8 + CAAM_PTR_SZ * 2, GFP_KERNEL | GFP_DMA); desc = kmalloc(CAAM_CMD_SZ * 8 + CAAM_PTR_SZ * 2, GFP_KERNEL | GFP_DMA);
@ -310,18 +412,9 @@ static int hash_digest_key(struct caam_hash_ctx *ctx, const u8 *key_in,
init_job_desc(desc, 0); init_job_desc(desc, 0);
src_dma = dma_map_single(jrdev, (void *)key_in, *keylen, key_dma = dma_map_single(jrdev, key, *keylen, DMA_BIDIRECTIONAL);
DMA_TO_DEVICE); if (dma_mapping_error(jrdev, key_dma)) {
if (dma_mapping_error(jrdev, src_dma)) { dev_err(jrdev, "unable to map key memory\n");
dev_err(jrdev, "unable to map key input memory\n");
kfree(desc);
return -ENOMEM;
}
dst_dma = dma_map_single(jrdev, (void *)key_out, digestsize,
DMA_FROM_DEVICE);
if (dma_mapping_error(jrdev, dst_dma)) {
dev_err(jrdev, "unable to map key output memory\n");
dma_unmap_single(jrdev, src_dma, *keylen, DMA_TO_DEVICE);
kfree(desc); kfree(desc);
return -ENOMEM; return -ENOMEM;
} }
@ -329,16 +422,16 @@ static int hash_digest_key(struct caam_hash_ctx *ctx, const u8 *key_in,
/* Job descriptor to perform unkeyed hash on key_in */ /* Job descriptor to perform unkeyed hash on key_in */
append_operation(desc, ctx->adata.algtype | OP_ALG_ENCRYPT | append_operation(desc, ctx->adata.algtype | OP_ALG_ENCRYPT |
OP_ALG_AS_INITFINAL); OP_ALG_AS_INITFINAL);
append_seq_in_ptr(desc, src_dma, *keylen, 0); append_seq_in_ptr(desc, key_dma, *keylen, 0);
append_seq_fifo_load(desc, *keylen, FIFOLD_CLASS_CLASS2 | append_seq_fifo_load(desc, *keylen, FIFOLD_CLASS_CLASS2 |
FIFOLD_TYPE_LAST2 | FIFOLD_TYPE_MSG); FIFOLD_TYPE_LAST2 | FIFOLD_TYPE_MSG);
append_seq_out_ptr(desc, dst_dma, digestsize, 0); append_seq_out_ptr(desc, key_dma, digestsize, 0);
append_seq_store(desc, digestsize, LDST_CLASS_2_CCB | append_seq_store(desc, digestsize, LDST_CLASS_2_CCB |
LDST_SRCDST_BYTE_CONTEXT); LDST_SRCDST_BYTE_CONTEXT);
#ifdef DEBUG #ifdef DEBUG
print_hex_dump(KERN_ERR, "key_in@"__stringify(__LINE__)": ", print_hex_dump(KERN_ERR, "key_in@"__stringify(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, key_in, *keylen, 1); DUMP_PREFIX_ADDRESS, 16, 4, key, *keylen, 1);
print_hex_dump(KERN_ERR, "jobdesc@"__stringify(__LINE__)": ", print_hex_dump(KERN_ERR, "jobdesc@"__stringify(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc), 1); DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc), 1);
#endif #endif
@ -354,12 +447,10 @@ static int hash_digest_key(struct caam_hash_ctx *ctx, const u8 *key_in,
#ifdef DEBUG #ifdef DEBUG
print_hex_dump(KERN_ERR, print_hex_dump(KERN_ERR,
"digested key@"__stringify(__LINE__)": ", "digested key@"__stringify(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, key_in, DUMP_PREFIX_ADDRESS, 16, 4, key, digestsize, 1);
digestsize, 1);
#endif #endif
} }
dma_unmap_single(jrdev, src_dma, *keylen, DMA_TO_DEVICE); dma_unmap_single(jrdev, key_dma, *keylen, DMA_BIDIRECTIONAL);
dma_unmap_single(jrdev, dst_dma, digestsize, DMA_FROM_DEVICE);
*keylen = digestsize; *keylen = digestsize;
@ -383,13 +474,10 @@ static int ahash_setkey(struct crypto_ahash *ahash,
#endif #endif
if (keylen > blocksize) { if (keylen > blocksize) {
hashed_key = kmalloc_array(digestsize, hashed_key = kmemdup(key, keylen, GFP_KERNEL | GFP_DMA);
sizeof(*hashed_key),
GFP_KERNEL | GFP_DMA);
if (!hashed_key) if (!hashed_key)
return -ENOMEM; return -ENOMEM;
ret = hash_digest_key(ctx, key, &keylen, hashed_key, ret = hash_digest_key(ctx, &keylen, hashed_key, digestsize);
digestsize);
if (ret) if (ret)
goto bad_free_key; goto bad_free_key;
key = hashed_key; key = hashed_key;
@ -424,9 +512,39 @@ static int ahash_setkey(struct crypto_ahash *ahash,
return -EINVAL; return -EINVAL;
} }
static int axcbc_setkey(struct crypto_ahash *ahash, const u8 *key,
unsigned int keylen)
{
struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
struct device *jrdev = ctx->jrdev;
memcpy(ctx->key, key, keylen);
dma_sync_single_for_device(jrdev, ctx->key_dma, keylen, DMA_TO_DEVICE);
ctx->adata.keylen = keylen;
print_hex_dump_debug("axcbc ctx.key@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, ctx->key, keylen, 1);
return axcbc_set_sh_desc(ahash);
}
static int acmac_setkey(struct crypto_ahash *ahash, const u8 *key,
unsigned int keylen)
{
struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
/* key is immediate data for all cmac shared descriptors */
ctx->adata.key_virt = key;
ctx->adata.keylen = keylen;
print_hex_dump_debug("acmac ctx.key@" __stringify(__LINE__)" : ",
DUMP_PREFIX_ADDRESS, 16, 4, key, keylen, 1);
return acmac_set_sh_desc(ahash);
}
/* /*
* ahash_edesc - s/w-extended ahash descriptor * ahash_edesc - s/w-extended ahash descriptor
* @dst_dma: physical mapped address of req->result
* @sec4_sg_dma: physical mapped address of h/w link table * @sec4_sg_dma: physical mapped address of h/w link table
* @src_nents: number of segments in input scatterlist * @src_nents: number of segments in input scatterlist
* @sec4_sg_bytes: length of dma mapped sec4_sg space * @sec4_sg_bytes: length of dma mapped sec4_sg space
@ -434,7 +552,6 @@ static int ahash_setkey(struct crypto_ahash *ahash,
* @sec4_sg: h/w link table * @sec4_sg: h/w link table
*/ */
struct ahash_edesc { struct ahash_edesc {
dma_addr_t dst_dma;
dma_addr_t sec4_sg_dma; dma_addr_t sec4_sg_dma;
int src_nents; int src_nents;
int sec4_sg_bytes; int sec4_sg_bytes;
@ -450,8 +567,6 @@ static inline void ahash_unmap(struct device *dev,
if (edesc->src_nents) if (edesc->src_nents)
dma_unmap_sg(dev, req->src, edesc->src_nents, DMA_TO_DEVICE); dma_unmap_sg(dev, req->src, edesc->src_nents, DMA_TO_DEVICE);
if (edesc->dst_dma)
dma_unmap_single(dev, edesc->dst_dma, dst_len, DMA_FROM_DEVICE);
if (edesc->sec4_sg_bytes) if (edesc->sec4_sg_bytes)
dma_unmap_single(dev, edesc->sec4_sg_dma, dma_unmap_single(dev, edesc->sec4_sg_dma,
@ -468,12 +583,10 @@ static inline void ahash_unmap_ctx(struct device *dev,
struct ahash_edesc *edesc, struct ahash_edesc *edesc,
struct ahash_request *req, int dst_len, u32 flag) struct ahash_request *req, int dst_len, u32 flag)
{ {
struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
struct caam_hash_state *state = ahash_request_ctx(req); struct caam_hash_state *state = ahash_request_ctx(req);
if (state->ctx_dma) { if (state->ctx_dma) {
dma_unmap_single(dev, state->ctx_dma, ctx->ctx_len, flag); dma_unmap_single(dev, state->ctx_dma, state->ctx_dma_len, flag);
state->ctx_dma = 0; state->ctx_dma = 0;
} }
ahash_unmap(dev, edesc, req, dst_len); ahash_unmap(dev, edesc, req, dst_len);
@ -486,9 +599,9 @@ static void ahash_done(struct device *jrdev, u32 *desc, u32 err,
struct ahash_edesc *edesc; struct ahash_edesc *edesc;
struct crypto_ahash *ahash = crypto_ahash_reqtfm(req); struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
int digestsize = crypto_ahash_digestsize(ahash); int digestsize = crypto_ahash_digestsize(ahash);
struct caam_hash_state *state = ahash_request_ctx(req);
#ifdef DEBUG #ifdef DEBUG
struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash); struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
struct caam_hash_state *state = ahash_request_ctx(req);
dev_err(jrdev, "%s %d: err 0x%x\n", __func__, __LINE__, err); dev_err(jrdev, "%s %d: err 0x%x\n", __func__, __LINE__, err);
#endif #endif
@ -497,17 +610,14 @@ static void ahash_done(struct device *jrdev, u32 *desc, u32 err,
if (err) if (err)
caam_jr_strstatus(jrdev, err); caam_jr_strstatus(jrdev, err);
ahash_unmap(jrdev, edesc, req, digestsize); ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_FROM_DEVICE);
memcpy(req->result, state->caam_ctx, digestsize);
kfree(edesc); kfree(edesc);
#ifdef DEBUG #ifdef DEBUG
print_hex_dump(KERN_ERR, "ctx@"__stringify(__LINE__)": ", print_hex_dump(KERN_ERR, "ctx@"__stringify(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, state->caam_ctx, DUMP_PREFIX_ADDRESS, 16, 4, state->caam_ctx,
ctx->ctx_len, 1); ctx->ctx_len, 1);
if (req->result)
print_hex_dump(KERN_ERR, "result@"__stringify(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, req->result,
digestsize, 1);
#endif #endif
req->base.complete(&req->base, err); req->base.complete(&req->base, err);
@ -555,9 +665,9 @@ static void ahash_done_ctx_src(struct device *jrdev, u32 *desc, u32 err,
struct ahash_edesc *edesc; struct ahash_edesc *edesc;
struct crypto_ahash *ahash = crypto_ahash_reqtfm(req); struct crypto_ahash *ahash = crypto_ahash_reqtfm(req);
int digestsize = crypto_ahash_digestsize(ahash); int digestsize = crypto_ahash_digestsize(ahash);
struct caam_hash_state *state = ahash_request_ctx(req);
#ifdef DEBUG #ifdef DEBUG
struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash); struct caam_hash_ctx *ctx = crypto_ahash_ctx(ahash);
struct caam_hash_state *state = ahash_request_ctx(req);
dev_err(jrdev, "%s %d: err 0x%x\n", __func__, __LINE__, err); dev_err(jrdev, "%s %d: err 0x%x\n", __func__, __LINE__, err);
#endif #endif
@ -566,17 +676,14 @@ static void ahash_done_ctx_src(struct device *jrdev, u32 *desc, u32 err,
if (err) if (err)
caam_jr_strstatus(jrdev, err); caam_jr_strstatus(jrdev, err);
ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_TO_DEVICE); ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_BIDIRECTIONAL);
memcpy(req->result, state->caam_ctx, digestsize);
kfree(edesc); kfree(edesc);
#ifdef DEBUG #ifdef DEBUG
print_hex_dump(KERN_ERR, "ctx@"__stringify(__LINE__)": ", print_hex_dump(KERN_ERR, "ctx@"__stringify(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, state->caam_ctx, DUMP_PREFIX_ADDRESS, 16, 4, state->caam_ctx,
ctx->ctx_len, 1); ctx->ctx_len, 1);
if (req->result)
print_hex_dump(KERN_ERR, "result@"__stringify(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, req->result,
digestsize, 1);
#endif #endif
req->base.complete(&req->base, err); req->base.complete(&req->base, err);
@ -688,6 +795,7 @@ static int ahash_update_ctx(struct ahash_request *req)
u8 *buf = current_buf(state); u8 *buf = current_buf(state);
int *buflen = current_buflen(state); int *buflen = current_buflen(state);
u8 *next_buf = alt_buf(state); u8 *next_buf = alt_buf(state);
int blocksize = crypto_ahash_blocksize(ahash);
int *next_buflen = alt_buflen(state), last_buflen; int *next_buflen = alt_buflen(state), last_buflen;
int in_len = *buflen + req->nbytes, to_hash; int in_len = *buflen + req->nbytes, to_hash;
u32 *desc; u32 *desc;
@ -696,9 +804,20 @@ static int ahash_update_ctx(struct ahash_request *req)
int ret = 0; int ret = 0;
last_buflen = *next_buflen; last_buflen = *next_buflen;
*next_buflen = in_len & (crypto_tfm_alg_blocksize(&ahash->base) - 1); *next_buflen = in_len & (blocksize - 1);
to_hash = in_len - *next_buflen; to_hash = in_len - *next_buflen;
/*
* For XCBC and CMAC, if to_hash is multiple of block size,
* keep last block in internal buffer
*/
if ((is_xcbc_aes(ctx->adata.algtype) ||
is_cmac_aes(ctx->adata.algtype)) && to_hash >= blocksize &&
(*next_buflen == 0)) {
*next_buflen = blocksize;
to_hash -= blocksize;
}
if (to_hash) { if (to_hash) {
src_nents = sg_nents_for_len(req->src, src_nents = sg_nents_for_len(req->src,
req->nbytes - (*next_buflen)); req->nbytes - (*next_buflen));
@ -801,7 +920,7 @@ static int ahash_update_ctx(struct ahash_request *req)
#endif #endif
return ret; return ret;
unmap_ctx: unmap_ctx:
ahash_unmap_ctx(jrdev, edesc, req, ctx->ctx_len, DMA_BIDIRECTIONAL); ahash_unmap_ctx(jrdev, edesc, req, ctx->ctx_len, DMA_BIDIRECTIONAL);
kfree(edesc); kfree(edesc);
return ret; return ret;
@ -837,7 +956,7 @@ static int ahash_final_ctx(struct ahash_request *req)
edesc->sec4_sg_bytes = sec4_sg_bytes; edesc->sec4_sg_bytes = sec4_sg_bytes;
ret = ctx_map_to_sec4_sg(jrdev, state, ctx->ctx_len, ret = ctx_map_to_sec4_sg(jrdev, state, ctx->ctx_len,
edesc->sec4_sg, DMA_TO_DEVICE); edesc->sec4_sg, DMA_BIDIRECTIONAL);
if (ret) if (ret)
goto unmap_ctx; goto unmap_ctx;
@ -857,14 +976,7 @@ static int ahash_final_ctx(struct ahash_request *req)
append_seq_in_ptr(desc, edesc->sec4_sg_dma, ctx->ctx_len + buflen, append_seq_in_ptr(desc, edesc->sec4_sg_dma, ctx->ctx_len + buflen,
LDST_SGF); LDST_SGF);
append_seq_out_ptr(desc, state->ctx_dma, digestsize, 0);
edesc->dst_dma = map_seq_out_ptr_result(desc, jrdev, req->result,
digestsize);
if (dma_mapping_error(jrdev, edesc->dst_dma)) {
dev_err(jrdev, "unable to map dst\n");
ret = -ENOMEM;
goto unmap_ctx;
}
#ifdef DEBUG #ifdef DEBUG
print_hex_dump(KERN_ERR, "jobdesc@"__stringify(__LINE__)": ", print_hex_dump(KERN_ERR, "jobdesc@"__stringify(__LINE__)": ",
@ -877,7 +989,7 @@ static int ahash_final_ctx(struct ahash_request *req)
return -EINPROGRESS; return -EINPROGRESS;
unmap_ctx: unmap_ctx:
ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_FROM_DEVICE); ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_BIDIRECTIONAL);
kfree(edesc); kfree(edesc);
return ret; return ret;
} }
@ -931,7 +1043,7 @@ static int ahash_finup_ctx(struct ahash_request *req)
edesc->src_nents = src_nents; edesc->src_nents = src_nents;
ret = ctx_map_to_sec4_sg(jrdev, state, ctx->ctx_len, ret = ctx_map_to_sec4_sg(jrdev, state, ctx->ctx_len,
edesc->sec4_sg, DMA_TO_DEVICE); edesc->sec4_sg, DMA_BIDIRECTIONAL);
if (ret) if (ret)
goto unmap_ctx; goto unmap_ctx;
@ -945,13 +1057,7 @@ static int ahash_finup_ctx(struct ahash_request *req)
if (ret) if (ret)
goto unmap_ctx; goto unmap_ctx;
edesc->dst_dma = map_seq_out_ptr_result(desc, jrdev, req->result, append_seq_out_ptr(desc, state->ctx_dma, digestsize, 0);
digestsize);
if (dma_mapping_error(jrdev, edesc->dst_dma)) {
dev_err(jrdev, "unable to map dst\n");
ret = -ENOMEM;
goto unmap_ctx;
}
#ifdef DEBUG #ifdef DEBUG
print_hex_dump(KERN_ERR, "jobdesc@"__stringify(__LINE__)": ", print_hex_dump(KERN_ERR, "jobdesc@"__stringify(__LINE__)": ",
@ -964,7 +1070,7 @@ static int ahash_finup_ctx(struct ahash_request *req)
return -EINPROGRESS; return -EINPROGRESS;
unmap_ctx: unmap_ctx:
ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_FROM_DEVICE); ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_BIDIRECTIONAL);
kfree(edesc); kfree(edesc);
return ret; return ret;
} }
@ -1023,10 +1129,8 @@ static int ahash_digest(struct ahash_request *req)
desc = edesc->hw_desc; desc = edesc->hw_desc;
edesc->dst_dma = map_seq_out_ptr_result(desc, jrdev, req->result, ret = map_seq_out_ptr_ctx(desc, jrdev, state, digestsize);
digestsize); if (ret) {
if (dma_mapping_error(jrdev, edesc->dst_dma)) {
dev_err(jrdev, "unable to map dst\n");
ahash_unmap(jrdev, edesc, req, digestsize); ahash_unmap(jrdev, edesc, req, digestsize);
kfree(edesc); kfree(edesc);
return -ENOMEM; return -ENOMEM;
@ -1041,7 +1145,7 @@ static int ahash_digest(struct ahash_request *req)
if (!ret) { if (!ret) {
ret = -EINPROGRESS; ret = -EINPROGRESS;
} else { } else {
ahash_unmap(jrdev, edesc, req, digestsize); ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_FROM_DEVICE);
kfree(edesc); kfree(edesc);
} }
@ -1083,12 +1187,9 @@ static int ahash_final_no_ctx(struct ahash_request *req)
append_seq_in_ptr(desc, state->buf_dma, buflen, 0); append_seq_in_ptr(desc, state->buf_dma, buflen, 0);
} }
edesc->dst_dma = map_seq_out_ptr_result(desc, jrdev, req->result, ret = map_seq_out_ptr_ctx(desc, jrdev, state, digestsize);
digestsize); if (ret)
if (dma_mapping_error(jrdev, edesc->dst_dma)) {
dev_err(jrdev, "unable to map dst\n");
goto unmap; goto unmap;
}
#ifdef DEBUG #ifdef DEBUG
print_hex_dump(KERN_ERR, "jobdesc@"__stringify(__LINE__)": ", print_hex_dump(KERN_ERR, "jobdesc@"__stringify(__LINE__)": ",
@ -1099,7 +1200,7 @@ static int ahash_final_no_ctx(struct ahash_request *req)
if (!ret) { if (!ret) {
ret = -EINPROGRESS; ret = -EINPROGRESS;
} else { } else {
ahash_unmap(jrdev, edesc, req, digestsize); ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_FROM_DEVICE);
kfree(edesc); kfree(edesc);
} }
@ -1122,6 +1223,7 @@ static int ahash_update_no_ctx(struct ahash_request *req)
GFP_KERNEL : GFP_ATOMIC; GFP_KERNEL : GFP_ATOMIC;
u8 *buf = current_buf(state); u8 *buf = current_buf(state);
int *buflen = current_buflen(state); int *buflen = current_buflen(state);
int blocksize = crypto_ahash_blocksize(ahash);
u8 *next_buf = alt_buf(state); u8 *next_buf = alt_buf(state);
int *next_buflen = alt_buflen(state); int *next_buflen = alt_buflen(state);
int in_len = *buflen + req->nbytes, to_hash; int in_len = *buflen + req->nbytes, to_hash;
@ -1130,9 +1232,20 @@ static int ahash_update_no_ctx(struct ahash_request *req)
u32 *desc; u32 *desc;
int ret = 0; int ret = 0;
*next_buflen = in_len & (crypto_tfm_alg_blocksize(&ahash->base) - 1); *next_buflen = in_len & (blocksize - 1);
to_hash = in_len - *next_buflen; to_hash = in_len - *next_buflen;
/*
* For XCBC and CMAC, if to_hash is multiple of block size,
* keep last block in internal buffer
*/
if ((is_xcbc_aes(ctx->adata.algtype) ||
is_cmac_aes(ctx->adata.algtype)) && to_hash >= blocksize &&
(*next_buflen == 0)) {
*next_buflen = blocksize;
to_hash -= blocksize;
}
if (to_hash) { if (to_hash) {
src_nents = sg_nents_for_len(req->src, src_nents = sg_nents_for_len(req->src,
req->nbytes - *next_buflen); req->nbytes - *next_buflen);
@ -1298,12 +1411,9 @@ static int ahash_finup_no_ctx(struct ahash_request *req)
goto unmap; goto unmap;
} }
edesc->dst_dma = map_seq_out_ptr_result(desc, jrdev, req->result, ret = map_seq_out_ptr_ctx(desc, jrdev, state, digestsize);
digestsize); if (ret)
if (dma_mapping_error(jrdev, edesc->dst_dma)) {
dev_err(jrdev, "unable to map dst\n");
goto unmap; goto unmap;
}
#ifdef DEBUG #ifdef DEBUG
print_hex_dump(KERN_ERR, "jobdesc@"__stringify(__LINE__)": ", print_hex_dump(KERN_ERR, "jobdesc@"__stringify(__LINE__)": ",
@ -1314,7 +1424,7 @@ static int ahash_finup_no_ctx(struct ahash_request *req)
if (!ret) { if (!ret) {
ret = -EINPROGRESS; ret = -EINPROGRESS;
} else { } else {
ahash_unmap(jrdev, edesc, req, digestsize); ahash_unmap_ctx(jrdev, edesc, req, digestsize, DMA_FROM_DEVICE);
kfree(edesc); kfree(edesc);
} }
@ -1338,15 +1448,26 @@ static int ahash_update_first(struct ahash_request *req)
u8 *next_buf = alt_buf(state); u8 *next_buf = alt_buf(state);
int *next_buflen = alt_buflen(state); int *next_buflen = alt_buflen(state);
int to_hash; int to_hash;
int blocksize = crypto_ahash_blocksize(ahash);
u32 *desc; u32 *desc;
int src_nents, mapped_nents; int src_nents, mapped_nents;
struct ahash_edesc *edesc; struct ahash_edesc *edesc;
int ret = 0; int ret = 0;
*next_buflen = req->nbytes & (crypto_tfm_alg_blocksize(&ahash->base) - *next_buflen = req->nbytes & (blocksize - 1);
1);
to_hash = req->nbytes - *next_buflen; to_hash = req->nbytes - *next_buflen;
/*
* For XCBC and CMAC, if to_hash is multiple of block size,
* keep last block in internal buffer
*/
if ((is_xcbc_aes(ctx->adata.algtype) ||
is_cmac_aes(ctx->adata.algtype)) && to_hash >= blocksize &&
(*next_buflen == 0)) {
*next_buflen = blocksize;
to_hash -= blocksize;
}
if (to_hash) { if (to_hash) {
src_nents = sg_nents_for_len(req->src, src_nents = sg_nents_for_len(req->src,
req->nbytes - *next_buflen); req->nbytes - *next_buflen);
@ -1446,6 +1567,7 @@ static int ahash_init(struct ahash_request *req)
state->final = ahash_final_no_ctx; state->final = ahash_final_no_ctx;
state->ctx_dma = 0; state->ctx_dma = 0;
state->ctx_dma_len = 0;
state->current_buf = 0; state->current_buf = 0;
state->buf_dma = 0; state->buf_dma = 0;
state->buflen_0 = 0; state->buflen_0 = 0;
@ -1654,6 +1776,44 @@ static struct caam_hash_template driver_hash[] = {
}, },
}, },
.alg_type = OP_ALG_ALGSEL_MD5, .alg_type = OP_ALG_ALGSEL_MD5,
}, {
.hmac_name = "xcbc(aes)",
.hmac_driver_name = "xcbc-aes-caam",
.blocksize = AES_BLOCK_SIZE,
.template_ahash = {
.init = ahash_init,
.update = ahash_update,
.final = ahash_final,
.finup = ahash_finup,
.digest = ahash_digest,
.export = ahash_export,
.import = ahash_import,
.setkey = axcbc_setkey,
.halg = {
.digestsize = AES_BLOCK_SIZE,
.statesize = sizeof(struct caam_export_state),
},
},
.alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_XCBC_MAC,
}, {
.hmac_name = "cmac(aes)",
.hmac_driver_name = "cmac-aes-caam",
.blocksize = AES_BLOCK_SIZE,
.template_ahash = {
.init = ahash_init,
.update = ahash_update,
.final = ahash_final,
.finup = ahash_finup,
.digest = ahash_digest,
.export = ahash_export,
.import = ahash_import,
.setkey = acmac_setkey,
.halg = {
.digestsize = AES_BLOCK_SIZE,
.statesize = sizeof(struct caam_export_state),
},
},
.alg_type = OP_ALG_ALGSEL_AES | OP_ALG_AAI_CMAC,
}, },
}; };
@ -1695,14 +1855,45 @@ static int caam_hash_cra_init(struct crypto_tfm *tfm)
} }
priv = dev_get_drvdata(ctx->jrdev->parent); priv = dev_get_drvdata(ctx->jrdev->parent);
ctx->dir = priv->era >= 6 ? DMA_BIDIRECTIONAL : DMA_TO_DEVICE;
if (is_xcbc_aes(caam_hash->alg_type)) {
ctx->dir = DMA_TO_DEVICE;
ctx->adata.algtype = OP_TYPE_CLASS1_ALG | caam_hash->alg_type;
ctx->ctx_len = 48;
ctx->key_dma = dma_map_single_attrs(ctx->jrdev, ctx->key,
ARRAY_SIZE(ctx->key),
DMA_BIDIRECTIONAL,
DMA_ATTR_SKIP_CPU_SYNC);
if (dma_mapping_error(ctx->jrdev, ctx->key_dma)) {
dev_err(ctx->jrdev, "unable to map key\n");
caam_jr_free(ctx->jrdev);
return -ENOMEM;
}
} else if (is_cmac_aes(caam_hash->alg_type)) {
ctx->dir = DMA_TO_DEVICE;
ctx->adata.algtype = OP_TYPE_CLASS1_ALG | caam_hash->alg_type;
ctx->ctx_len = 32;
} else {
ctx->dir = priv->era >= 6 ? DMA_BIDIRECTIONAL : DMA_TO_DEVICE;
ctx->adata.algtype = OP_TYPE_CLASS2_ALG | caam_hash->alg_type;
ctx->ctx_len = runninglen[(ctx->adata.algtype &
OP_ALG_ALGSEL_SUBMASK) >>
OP_ALG_ALGSEL_SHIFT];
}
dma_addr = dma_map_single_attrs(ctx->jrdev, ctx->sh_desc_update, dma_addr = dma_map_single_attrs(ctx->jrdev, ctx->sh_desc_update,
offsetof(struct caam_hash_ctx, offsetof(struct caam_hash_ctx, key),
sh_desc_update_dma),
ctx->dir, DMA_ATTR_SKIP_CPU_SYNC); ctx->dir, DMA_ATTR_SKIP_CPU_SYNC);
if (dma_mapping_error(ctx->jrdev, dma_addr)) { if (dma_mapping_error(ctx->jrdev, dma_addr)) {
dev_err(ctx->jrdev, "unable to map shared descriptors\n"); dev_err(ctx->jrdev, "unable to map shared descriptors\n");
if (is_xcbc_aes(caam_hash->alg_type))
dma_unmap_single_attrs(ctx->jrdev, ctx->key_dma,
ARRAY_SIZE(ctx->key),
DMA_BIDIRECTIONAL,
DMA_ATTR_SKIP_CPU_SYNC);
caam_jr_free(ctx->jrdev); caam_jr_free(ctx->jrdev);
return -ENOMEM; return -ENOMEM;
} }
@ -1716,16 +1907,14 @@ static int caam_hash_cra_init(struct crypto_tfm *tfm)
ctx->sh_desc_digest_dma = dma_addr + offsetof(struct caam_hash_ctx, ctx->sh_desc_digest_dma = dma_addr + offsetof(struct caam_hash_ctx,
sh_desc_digest); sh_desc_digest);
/* copy descriptor header template value */
ctx->adata.algtype = OP_TYPE_CLASS2_ALG | caam_hash->alg_type;
ctx->ctx_len = runninglen[(ctx->adata.algtype &
OP_ALG_ALGSEL_SUBMASK) >>
OP_ALG_ALGSEL_SHIFT];
crypto_ahash_set_reqsize(__crypto_ahash_cast(tfm), crypto_ahash_set_reqsize(__crypto_ahash_cast(tfm),
sizeof(struct caam_hash_state)); sizeof(struct caam_hash_state));
return ahash_set_sh_desc(ahash);
/*
* For keyed hash algorithms shared descriptors
* will be created later in setkey() callback
*/
return alg->setkey ? 0 : ahash_set_sh_desc(ahash);
} }
static void caam_hash_cra_exit(struct crypto_tfm *tfm) static void caam_hash_cra_exit(struct crypto_tfm *tfm)
@ -1733,9 +1922,12 @@ static void caam_hash_cra_exit(struct crypto_tfm *tfm)
struct caam_hash_ctx *ctx = crypto_tfm_ctx(tfm); struct caam_hash_ctx *ctx = crypto_tfm_ctx(tfm);
dma_unmap_single_attrs(ctx->jrdev, ctx->sh_desc_update_dma, dma_unmap_single_attrs(ctx->jrdev, ctx->sh_desc_update_dma,
offsetof(struct caam_hash_ctx, offsetof(struct caam_hash_ctx, key),
sh_desc_update_dma),
ctx->dir, DMA_ATTR_SKIP_CPU_SYNC); ctx->dir, DMA_ATTR_SKIP_CPU_SYNC);
if (is_xcbc_aes(ctx->adata.algtype))
dma_unmap_single_attrs(ctx->jrdev, ctx->key_dma,
ARRAY_SIZE(ctx->key), DMA_BIDIRECTIONAL,
DMA_ATTR_SKIP_CPU_SYNC);
caam_jr_free(ctx->jrdev); caam_jr_free(ctx->jrdev);
} }
@ -1866,14 +2058,16 @@ static int __init caam_algapi_hash_init(void)
struct caam_hash_template *alg = driver_hash + i; struct caam_hash_template *alg = driver_hash + i;
/* If MD size is not supported by device, skip registration */ /* If MD size is not supported by device, skip registration */
if (alg->template_ahash.halg.digestsize > md_limit) if (is_mdha(alg->alg_type) &&
alg->template_ahash.halg.digestsize > md_limit)
continue; continue;
/* register hmac version */ /* register hmac version */
t_alg = caam_hash_alloc(alg, true); t_alg = caam_hash_alloc(alg, true);
if (IS_ERR(t_alg)) { if (IS_ERR(t_alg)) {
err = PTR_ERR(t_alg); err = PTR_ERR(t_alg);
pr_warn("%s alg allocation failed\n", alg->driver_name); pr_warn("%s alg allocation failed\n",
alg->hmac_driver_name);
continue; continue;
} }
@ -1886,6 +2080,9 @@ static int __init caam_algapi_hash_init(void)
} else } else
list_add_tail(&t_alg->entry, &hash_list); list_add_tail(&t_alg->entry, &hash_list);
if ((alg->alg_type & OP_ALG_ALGSEL_MASK) == OP_ALG_ALGSEL_AES)
continue;
/* register unkeyed version */ /* register unkeyed version */
t_alg = caam_hash_alloc(alg, false); t_alg = caam_hash_alloc(alg, false);
if (IS_ERR(t_alg)) { if (IS_ERR(t_alg)) {

68
drivers/crypto/caam/caamhash_desc.c
View File

@ -2,7 +2,7 @@
/* /*
* Shared descriptors for ahash algorithms * Shared descriptors for ahash algorithms
* *
* Copyright 2017 NXP * Copyright 2017-2019 NXP
*/ */
#include "compat.h" #include "compat.h"
@ -75,6 +75,72 @@ void cnstr_shdsc_ahash(u32 * const desc, struct alginfo *adata, u32 state,
} }
EXPORT_SYMBOL(cnstr_shdsc_ahash); EXPORT_SYMBOL(cnstr_shdsc_ahash);
/**
* cnstr_shdsc_sk_hash - shared descriptor for symmetric key cipher-based
* hash algorithms
* @desc: pointer to buffer used for descriptor construction
* @adata: pointer to authentication transform definitions.
* @state: algorithm state OP_ALG_AS_{INIT, FINALIZE, INITFINALIZE, UPDATE}
* @digestsize: algorithm's digest size
* @ctx_len: size of Context Register
* @key_dma: I/O Virtual Address of the key
*/
void cnstr_shdsc_sk_hash(u32 * const desc, struct alginfo *adata, u32 state,
int digestsize, int ctx_len, dma_addr_t key_dma)
{
u32 *skip_key_load;
init_sh_desc(desc, HDR_SHARE_SERIAL | HDR_SAVECTX);
/* Skip loading of key, context if already shared */
skip_key_load = append_jump(desc, JUMP_TEST_ALL | JUMP_COND_SHRD);
if (state == OP_ALG_AS_INIT || state == OP_ALG_AS_INITFINAL) {
append_key_as_imm(desc, adata->key_virt, adata->keylen,
adata->keylen, CLASS_1 | KEY_DEST_CLASS_REG);
} else { /* UPDATE, FINALIZE */
if (is_xcbc_aes(adata->algtype))
/* Load K1 */
append_key(desc, adata->key_dma, adata->keylen,
CLASS_1 | KEY_DEST_CLASS_REG | KEY_ENC);
else /* CMAC */
append_key_as_imm(desc, adata->key_virt, adata->keylen,
adata->keylen, CLASS_1 |
KEY_DEST_CLASS_REG);
/* Restore context */
append_seq_load(desc, ctx_len, LDST_CLASS_1_CCB |
LDST_SRCDST_BYTE_CONTEXT);
}
set_jump_tgt_here(desc, skip_key_load);
/* Class 1 operation */
append_operation(desc, adata->algtype | state | OP_ALG_ENCRYPT);
/*
* Load from buf and/or src and write to req->result or state->context
* Calculate remaining bytes to read
*/
append_math_add(desc, VARSEQINLEN, SEQINLEN, REG0, CAAM_CMD_SZ);
/* Read remaining bytes */
append_seq_fifo_load(desc, 0, FIFOLD_CLASS_CLASS1 | FIFOLD_TYPE_LAST1 |
FIFOLD_TYPE_MSG | FIFOLDST_VLF);
/*
* Save context:
* - xcbc: partial hash, keys K2 and K3
* - cmac: partial hash, constant L = E(K,0)
*/
append_seq_store(desc, digestsize, LDST_CLASS_1_CCB |
LDST_SRCDST_BYTE_CONTEXT);
if (is_xcbc_aes(adata->algtype) && state == OP_ALG_AS_INIT)
/* Save K1 */
append_fifo_store(desc, key_dma, adata->keylen,
LDST_CLASS_1_CCB | FIFOST_TYPE_KEY_KEK);
}
EXPORT_SYMBOL(cnstr_shdsc_sk_hash);
MODULE_LICENSE("Dual BSD/GPL"); MODULE_LICENSE("Dual BSD/GPL");
MODULE_DESCRIPTION("FSL CAAM ahash descriptors support"); MODULE_DESCRIPTION("FSL CAAM ahash descriptors support");
MODULE_AUTHOR("NXP Semiconductors"); MODULE_AUTHOR("NXP Semiconductors");

8
drivers/crypto/caam/caamhash_desc.h
View File

@ -15,7 +15,15 @@
#define DESC_AHASH_FINAL_LEN (DESC_AHASH_BASE + 5 * CAAM_CMD_SZ) #define DESC_AHASH_FINAL_LEN (DESC_AHASH_BASE + 5 * CAAM_CMD_SZ)
#define DESC_AHASH_DIGEST_LEN (DESC_AHASH_BASE + 4 * CAAM_CMD_SZ) #define DESC_AHASH_DIGEST_LEN (DESC_AHASH_BASE + 4 * CAAM_CMD_SZ)
static inline bool is_xcbc_aes(u32 algtype)
{
return (algtype & (OP_ALG_ALGSEL_MASK | OP_ALG_AAI_MASK)) ==
(OP_ALG_ALGSEL_AES | OP_ALG_AAI_XCBC_MAC);
}
void cnstr_shdsc_ahash(u32 * const desc, struct alginfo *adata, u32 state, void cnstr_shdsc_ahash(u32 * const desc, struct alginfo *adata, u32 state,
int digestsize, int ctx_len, bool import_ctx, int era); int digestsize, int ctx_len, bool import_ctx, int era);
void cnstr_shdsc_sk_hash(u32 * const desc, struct alginfo *adata, u32 state,
int digestsize, int ctx_len, dma_addr_t key_dma);
#endif /* _CAAMHASH_DESC_H_ */ #endif /* _CAAMHASH_DESC_H_ */

1
drivers/crypto/caam/compat.h
View File

@ -43,6 +43,7 @@
#include <crypto/akcipher.h> #include <crypto/akcipher.h>
#include <crypto/scatterwalk.h> #include <crypto/scatterwalk.h>
#include <crypto/skcipher.h> #include <crypto/skcipher.h>
#include <crypto/arc4.h>
#include <crypto/internal/skcipher.h> #include <crypto/internal/skcipher.h>
#include <crypto/internal/hash.h> #include <crypto/internal/hash.h>
#include <crypto/internal/rsa.h> #include <crypto/internal/rsa.h>

25
drivers/crypto/caam/ctrl.c
View File

@ -18,12 +18,8 @@
#include "desc_constr.h" #include "desc_constr.h"
#include "ctrl.h" #include "ctrl.h"
bool caam_little_end;
EXPORT_SYMBOL(caam_little_end);
bool caam_dpaa2; bool caam_dpaa2;
EXPORT_SYMBOL(caam_dpaa2); EXPORT_SYMBOL(caam_dpaa2);
bool caam_imx;
EXPORT_SYMBOL(caam_imx);
#ifdef CONFIG_CAAM_QI #ifdef CONFIG_CAAM_QI
#include "qi.h" #include "qi.h"
@ -863,27 +859,18 @@ static int caam_probe(struct platform_device *pdev)
/* Internal covering keys (useful in non-secure mode only) */ /* Internal covering keys (useful in non-secure mode only) */
ctrlpriv->ctl_kek_wrap.data = (__force void *)&ctrlpriv->ctrl->kek[0]; ctrlpriv->ctl_kek_wrap.data = (__force void *)&ctrlpriv->ctrl->kek[0];
ctrlpriv->ctl_kek_wrap.size = KEK_KEY_SIZE * sizeof(u32); ctrlpriv->ctl_kek_wrap.size = KEK_KEY_SIZE * sizeof(u32);
ctrlpriv->ctl_kek = debugfs_create_blob("kek", debugfs_create_blob("kek", S_IRUSR | S_IRGRP | S_IROTH, ctrlpriv->ctl,
S_IRUSR | &ctrlpriv->ctl_kek_wrap);
S_IRGRP | S_IROTH,
ctrlpriv->ctl,
&ctrlpriv->ctl_kek_wrap);
ctrlpriv->ctl_tkek_wrap.data = (__force void *)&ctrlpriv->ctrl->tkek[0]; ctrlpriv->ctl_tkek_wrap.data = (__force void *)&ctrlpriv->ctrl->tkek[0];
ctrlpriv->ctl_tkek_wrap.size = KEK_KEY_SIZE * sizeof(u32); ctrlpriv->ctl_tkek_wrap.size = KEK_KEY_SIZE * sizeof(u32);
ctrlpriv->ctl_tkek = debugfs_create_blob("tkek", debugfs_create_blob("tkek", S_IRUSR | S_IRGRP | S_IROTH, ctrlpriv->ctl,
S_IRUSR | &ctrlpriv->ctl_tkek_wrap);
S_IRGRP | S_IROTH,
ctrlpriv->ctl,
&ctrlpriv->ctl_tkek_wrap);
ctrlpriv->ctl_tdsk_wrap.data = (__force void *)&ctrlpriv->ctrl->tdsk[0]; ctrlpriv->ctl_tdsk_wrap.data = (__force void *)&ctrlpriv->ctrl->tdsk[0];
ctrlpriv->ctl_tdsk_wrap.size = KEK_KEY_SIZE * sizeof(u32); ctrlpriv->ctl_tdsk_wrap.size = KEK_KEY_SIZE * sizeof(u32);
ctrlpriv->ctl_tdsk = debugfs_create_blob("tdsk", debugfs_create_blob("tdsk", S_IRUSR | S_IRGRP | S_IROTH, ctrlpriv->ctl,
S_IRUSR | &ctrlpriv->ctl_tdsk_wrap);
S_IRGRP | S_IROTH,
ctrlpriv->ctl,
&ctrlpriv->ctl_tdsk_wrap);
#endif #endif
return 0; return 0;

6
drivers/crypto/caam/error.c
View File

@ -50,6 +50,12 @@ void caam_dump_sg(const char *level, const char *prefix_str, int prefix_type,
#endif /* DEBUG */ #endif /* DEBUG */
EXPORT_SYMBOL(caam_dump_sg); EXPORT_SYMBOL(caam_dump_sg);
bool caam_little_end;
EXPORT_SYMBOL(caam_little_end);
bool caam_imx;
EXPORT_SYMBOL(caam_imx);
static const struct { static const struct {
u8 value; u8 value;
const char *error_text; const char *error_text;

1
drivers/crypto/caam/intern.h
View File

@ -106,7 +106,6 @@ struct caam_drv_private {
struct dentry *dfs_root; struct dentry *dfs_root;
struct dentry *ctl; /* controller dir */ struct dentry *ctl; /* controller dir */
struct debugfs_blob_wrapper ctl_kek_wrap, ctl_tkek_wrap, ctl_tdsk_wrap; struct debugfs_blob_wrapper ctl_kek_wrap, ctl_tkek_wrap, ctl_tdsk_wrap;
struct dentry *ctl_kek, *ctl_tkek, *ctl_tdsk;
#endif #endif
}; };

30
drivers/crypto/caam/key_gen.c
View File

@ -48,7 +48,7 @@ int gen_split_key(struct device *jrdev, u8 *key_out,
{ {
u32 *desc; u32 *desc;
struct split_key_result result; struct split_key_result result;
dma_addr_t dma_addr_in, dma_addr_out; dma_addr_t dma_addr;
int ret = -ENOMEM; int ret = -ENOMEM;
adata->keylen = split_key_len(adata->algtype & OP_ALG_ALGSEL_MASK); adata->keylen = split_key_len(adata->algtype & OP_ALG_ALGSEL_MASK);
@ -71,22 +71,17 @@ int gen_split_key(struct device *jrdev, u8 *key_out,
return ret; return ret;
} }
dma_addr_in = dma_map_single(jrdev, (void *)key_in, keylen, memcpy(key_out, key_in, keylen);
DMA_TO_DEVICE);
if (dma_mapping_error(jrdev, dma_addr_in)) { dma_addr = dma_map_single(jrdev, key_out, adata->keylen_pad,
dev_err(jrdev, "unable to map key input memory\n"); DMA_BIDIRECTIONAL);
if (dma_mapping_error(jrdev, dma_addr)) {
dev_err(jrdev, "unable to map key memory\n");
goto out_free; goto out_free;
} }
dma_addr_out = dma_map_single(jrdev, key_out, adata->keylen_pad,
DMA_FROM_DEVICE);
if (dma_mapping_error(jrdev, dma_addr_out)) {
dev_err(jrdev, "unable to map key output memory\n");
goto out_unmap_in;
}
init_job_desc(desc, 0); init_job_desc(desc, 0);
append_key(desc, dma_addr_in, keylen, CLASS_2 | KEY_DEST_CLASS_REG); append_key(desc, dma_addr, keylen, CLASS_2 | KEY_DEST_CLASS_REG);
/* Sets MDHA up into an HMAC-INIT */ /* Sets MDHA up into an HMAC-INIT */
append_operation(desc, (adata->algtype & OP_ALG_ALGSEL_MASK) | append_operation(desc, (adata->algtype & OP_ALG_ALGSEL_MASK) |
@ -104,12 +99,10 @@ int gen_split_key(struct device *jrdev, u8 *key_out,
* FIFO_STORE with the explicit split-key content store * FIFO_STORE with the explicit split-key content store
* (0x26 output type) * (0x26 output type)
*/ */
append_fifo_store(desc, dma_addr_out, adata->keylen, append_fifo_store(desc, dma_addr, adata->keylen,
LDST_CLASS_2_CCB | FIFOST_TYPE_SPLIT_KEK); LDST_CLASS_2_CCB | FIFOST_TYPE_SPLIT_KEK);
#ifdef DEBUG #ifdef DEBUG
print_hex_dump(KERN_ERR, "ctx.key@"__stringify(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, key_in, keylen, 1);
print_hex_dump(KERN_ERR, "jobdesc@"__stringify(__LINE__)": ", print_hex_dump(KERN_ERR, "jobdesc@"__stringify(__LINE__)": ",
DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc), 1); DUMP_PREFIX_ADDRESS, 16, 4, desc, desc_bytes(desc), 1);
#endif #endif
@ -129,10 +122,7 @@ int gen_split_key(struct device *jrdev, u8 *key_out,
#endif #endif
} }
dma_unmap_single(jrdev, dma_addr_out, adata->keylen_pad, dma_unmap_single(jrdev, dma_addr, adata->keylen_pad, DMA_BIDIRECTIONAL);
DMA_FROM_DEVICE);
out_unmap_in:
dma_unmap_single(jrdev, dma_addr_in, keylen, DMA_TO_DEVICE);
out_free: out_free:
kfree(desc); kfree(desc);
return ret; return ret;

4
drivers/crypto/caam/qi.c
View File

@ -318,7 +318,7 @@ int caam_drv_ctx_update(struct caam_drv_ctx *drv_ctx, u32 *sh_desc)
/* Create a new req FQ in parked state */ /* Create a new req FQ in parked state */
new_fq = create_caam_req_fq(drv_ctx->qidev, drv_ctx->rsp_fq, new_fq = create_caam_req_fq(drv_ctx->qidev, drv_ctx->rsp_fq,
drv_ctx->context_a, 0); drv_ctx->context_a, 0);
if (unlikely(IS_ERR_OR_NULL(new_fq))) { if (IS_ERR_OR_NULL(new_fq)) {
dev_err(qidev, "FQ allocation for shdesc update failed\n"); dev_err(qidev, "FQ allocation for shdesc update failed\n");
return PTR_ERR(new_fq); return PTR_ERR(new_fq);
} }
@ -431,7 +431,7 @@ struct caam_drv_ctx *caam_drv_ctx_init(struct device *qidev,
/* Attach request FQ */ /* Attach request FQ */
drv_ctx->req_fq = create_caam_req_fq(qidev, drv_ctx->rsp_fq, hwdesc, drv_ctx->req_fq = create_caam_req_fq(qidev, drv_ctx->rsp_fq, hwdesc,
QMAN_INITFQ_FLAG_SCHED); QMAN_INITFQ_FLAG_SCHED);
if (unlikely(IS_ERR_OR_NULL(drv_ctx->req_fq))) { if (IS_ERR_OR_NULL(drv_ctx->req_fq)) {
dev_err(qidev, "create_caam_req_fq failed\n"); dev_err(qidev, "create_caam_req_fq failed\n");
dma_unmap_single(qidev, hwdesc, size, DMA_BIDIRECTIONAL); dma_unmap_single(qidev, hwdesc, size, DMA_BIDIRECTIONAL);
kfree(drv_ctx); kfree(drv_ctx);

27
drivers/crypto/cavium/nitrox/nitrox_debugfs.c
View File

@ -55,31 +55,14 @@ void nitrox_debugfs_exit(struct nitrox_device *ndev)
ndev->debugfs_dir = NULL; ndev->debugfs_dir = NULL;
} }
int nitrox_debugfs_init(struct nitrox_device *ndev) void nitrox_debugfs_init(struct nitrox_device *ndev)
{ {
struct dentry *dir, *f; struct dentry *dir;
dir = debugfs_create_dir(KBUILD_MODNAME, NULL); dir = debugfs_create_dir(KBUILD_MODNAME, NULL);
if (!dir)
return -ENOMEM;
ndev->debugfs_dir = dir; ndev->debugfs_dir = dir;
f = debugfs_create_file("firmware", 0400, dir, ndev, debugfs_create_file("firmware", 0400, dir, ndev, &firmware_fops);
&firmware_fops); debugfs_create_file("device", 0400, dir, ndev, &device_fops);
if (!f) debugfs_create_file("stats", 0400, dir, ndev, &stats_fops);
goto err;
f = debugfs_create_file("device", 0400, dir, ndev,
&device_fops);
if (!f)
goto err;
f = debugfs_create_file("stats", 0400, dir, ndev,
&stats_fops);
if (!f)
goto err;
return 0;
err:
nitrox_debugfs_exit(ndev);
return -ENODEV;
} }

5
drivers/crypto/cavium/nitrox/nitrox_debugfs.h
View File

@ -5,12 +5,11 @@
#include "nitrox_dev.h" #include "nitrox_dev.h"
#ifdef CONFIG_DEBUG_FS #ifdef CONFIG_DEBUG_FS
int nitrox_debugfs_init(struct nitrox_device *ndev); void nitrox_debugfs_init(struct nitrox_device *ndev);
void nitrox_debugfs_exit(struct nitrox_device *ndev); void nitrox_debugfs_exit(struct nitrox_device *ndev);
#else #else
static inline int nitrox_debugfs_init(struct nitrox_device *ndev) static inline void nitrox_debugfs_init(struct nitrox_device *ndev)
{ {
return 0;
} }
static inline void nitrox_debugfs_exit(struct nitrox_device *ndev) static inline void nitrox_debugfs_exit(struct nitrox_device *ndev)

4
drivers/crypto/cavium/nitrox/nitrox_main.c
View File

@ -404,9 +404,7 @@ static int nitrox_probe(struct pci_dev *pdev,
if (err) if (err)
goto pf_hw_fail; goto pf_hw_fail;
err = nitrox_debugfs_init(ndev); nitrox_debugfs_init(ndev);
if (err)
goto pf_hw_fail;
/* clear the statistics */ /* clear the statistics */
atomic64_set(&ndev->stats.posted, 0); atomic64_set(&ndev->stats.posted, 0);

56
drivers/crypto/cavium/zip/zip_main.c
View File

@ -351,6 +351,7 @@ static struct pci_driver zip_driver = {
static struct crypto_alg zip_comp_deflate = { static struct crypto_alg zip_comp_deflate = {
.cra_name = "deflate", .cra_name = "deflate",
.cra_driver_name = "deflate-cavium",
.cra_flags = CRYPTO_ALG_TYPE_COMPRESS, .cra_flags = CRYPTO_ALG_TYPE_COMPRESS,
.cra_ctxsize = sizeof(struct zip_kernel_ctx), .cra_ctxsize = sizeof(struct zip_kernel_ctx),
.cra_priority = 300, .cra_priority = 300,
@ -365,6 +366,7 @@ static struct crypto_alg zip_comp_deflate = {
static struct crypto_alg zip_comp_lzs = { static struct crypto_alg zip_comp_lzs = {
.cra_name = "lzs", .cra_name = "lzs",
.cra_driver_name = "lzs-cavium",
.cra_flags = CRYPTO_ALG_TYPE_COMPRESS, .cra_flags = CRYPTO_ALG_TYPE_COMPRESS,
.cra_ctxsize = sizeof(struct zip_kernel_ctx), .cra_ctxsize = sizeof(struct zip_kernel_ctx),
.cra_priority = 300, .cra_priority = 300,
@ -384,7 +386,7 @@ static struct scomp_alg zip_scomp_deflate = {
.decompress = zip_scomp_decompress, .decompress = zip_scomp_decompress,
.base = { .base = {
.cra_name = "deflate", .cra_name = "deflate",
.cra_driver_name = "deflate-scomp", .cra_driver_name = "deflate-scomp-cavium",
.cra_module = THIS_MODULE, .cra_module = THIS_MODULE,
.cra_priority = 300, .cra_priority = 300,
} }
@ -397,7 +399,7 @@ static struct scomp_alg zip_scomp_lzs = {
.decompress = zip_scomp_decompress, .decompress = zip_scomp_decompress,
.base = { .base = {
.cra_name = "lzs", .cra_name = "lzs",
.cra_driver_name = "lzs-scomp", .cra_driver_name = "lzs-scomp-cavium",
.cra_module = THIS_MODULE, .cra_module = THIS_MODULE,
.cra_priority = 300, .cra_priority = 300,
} }
@ -618,41 +620,23 @@ static const struct file_operations zip_regs_fops = {
/* Root directory for thunderx_zip debugfs entry */ /* Root directory for thunderx_zip debugfs entry */
static struct dentry *zip_debugfs_root; static struct dentry *zip_debugfs_root;
static int __init zip_debugfs_init(void) static void __init zip_debugfs_init(void)
{ {
struct dentry *zip_stats, *zip_clear, *zip_regs;
if (!debugfs_initialized()) if (!debugfs_initialized())
return -ENODEV; return;
zip_debugfs_root = debugfs_create_dir("thunderx_zip", NULL); zip_debugfs_root = debugfs_create_dir("thunderx_zip", NULL);
if (!zip_debugfs_root)
return -ENOMEM;
/* Creating files for entries inside thunderx_zip directory */ /* Creating files for entries inside thunderx_zip directory */
zip_stats = debugfs_create_file("zip_stats", 0444, debugfs_create_file("zip_stats", 0444, zip_debugfs_root, NULL,
zip_debugfs_root, &zip_stats_fops);
NULL, &zip_stats_fops);
if (!zip_stats)
goto failed_to_create;
zip_clear = debugfs_create_file("zip_clear", 0444, debugfs_create_file("zip_clear", 0444, zip_debugfs_root, NULL,
zip_debugfs_root, &zip_clear_fops);
NULL, &zip_clear_fops);
if (!zip_clear)
goto failed_to_create;
zip_regs = debugfs_create_file("zip_regs", 0444, debugfs_create_file("zip_regs", 0444, zip_debugfs_root, NULL,
zip_debugfs_root, &zip_regs_fops);
NULL, &zip_regs_fops);
if (!zip_regs)
goto failed_to_create;
return 0;
failed_to_create:
debugfs_remove_recursive(zip_debugfs_root);
return -ENOENT;
} }
static void __exit zip_debugfs_exit(void) static void __exit zip_debugfs_exit(void)
@ -661,13 +645,8 @@ static void __exit zip_debugfs_exit(void)
} }
#else #else
static int __init zip_debugfs_init(void) static void __init zip_debugfs_init(void) { }
{
return 0;
}
static void __exit zip_debugfs_exit(void) { } static void __exit zip_debugfs_exit(void) { }
#endif #endif
/* debugfs - end */ /* debugfs - end */
@ -691,17 +670,10 @@ static int __init zip_init_module(void)
} }
/* comp-decomp statistics are handled with debugfs interface */ /* comp-decomp statistics are handled with debugfs interface */
ret = zip_debugfs_init(); zip_debugfs_init();
if (ret < 0) {
zip_err("ZIP: debugfs initialization failed\n");
goto err_crypto_unregister;
}
return ret; return ret;
err_crypto_unregister:
zip_unregister_compression_device();
err_pci_unregister: err_pci_unregister:
pci_unregister_driver(&zip_driver); pci_unregister_driver(&zip_driver);
return ret; return ret;

2
drivers/crypto/ccp/ccp-crypto-aes-cmac.c
View File

@ -1,7 +1,7 @@
/* /*
* AMD Cryptographic Coprocessor (CCP) AES CMAC crypto API support * AMD Cryptographic Coprocessor (CCP) AES CMAC crypto API support
* *
* Copyright (C) 2013 Advanced Micro Devices, Inc. * Copyright (C) 2013,2018 Advanced Micro Devices, Inc.
* *
* Author: Tom Lendacky <thomas.lendacky@amd.com> * Author: Tom Lendacky <thomas.lendacky@amd.com>
* *

2
drivers/crypto/ccp/ccp-crypto-des3.c
View File

@ -57,7 +57,7 @@ static int ccp_des3_setkey(struct crypto_ablkcipher *tfm, const u8 *key,
if (unlikely(!((K[0] ^ K[2]) | (K[1] ^ K[3])) || if (unlikely(!((K[0] ^ K[2]) | (K[1] ^ K[3])) ||
!((K[2] ^ K[4]) | (K[3] ^ K[5]))) && !((K[2] ^ K[4]) | (K[3] ^ K[5]))) &&
(*flags & CRYPTO_TFM_REQ_WEAK_KEY)) { (*flags & CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) {
*flags |= CRYPTO_TFM_RES_WEAK_KEY; *flags |= CRYPTO_TFM_RES_WEAK_KEY;
return -EINVAL; return -EINVAL;
} }

2
drivers/crypto/ccp/ccp-crypto-sha.c
View File

@ -1,7 +1,7 @@
/* /*
* AMD Cryptographic Coprocessor (CCP) SHA crypto API support * AMD Cryptographic Coprocessor (CCP) SHA crypto API support
* *
* Copyright (C) 2013,2017 Advanced Micro Devices, Inc. * Copyright (C) 2013,2018 Advanced Micro Devices, Inc.
* *
* Author: Tom Lendacky <thomas.lendacky@amd.com> * Author: Tom Lendacky <thomas.lendacky@amd.com>
* Author: Gary R Hook <gary.hook@amd.com> * Author: Gary R Hook <gary.hook@amd.com>

34
drivers/crypto/ccp/ccp-debugfs.c
View File

@ -286,10 +286,7 @@ void ccp5_debugfs_setup(struct ccp_device *ccp)
{ {
struct ccp_cmd_queue *cmd_q; struct ccp_cmd_queue *cmd_q;
char name[MAX_NAME_LEN + 1]; char name[MAX_NAME_LEN + 1];
struct dentry *debugfs_info;
struct dentry *debugfs_stats;
struct dentry *debugfs_q_instance; struct dentry *debugfs_q_instance;
struct dentry *debugfs_q_stats;
int i; int i;
if (!debugfs_initialized()) if (!debugfs_initialized())
@ -299,24 +296,14 @@ void ccp5_debugfs_setup(struct ccp_device *ccp)
if (!ccp_debugfs_dir) if (!ccp_debugfs_dir)
ccp_debugfs_dir = debugfs_create_dir(KBUILD_MODNAME, NULL); ccp_debugfs_dir = debugfs_create_dir(KBUILD_MODNAME, NULL);
mutex_unlock(&ccp_debugfs_lock); mutex_unlock(&ccp_debugfs_lock);
if (!ccp_debugfs_dir)
return;
ccp->debugfs_instance = debugfs_create_dir(ccp->name, ccp_debugfs_dir); ccp->debugfs_instance = debugfs_create_dir(ccp->name, ccp_debugfs_dir);
if (!ccp->debugfs_instance)
goto err;
debugfs_info = debugfs_create_file("info", 0400, debugfs_create_file("info", 0400, ccp->debugfs_instance, ccp,
ccp->debugfs_instance, ccp, &ccp_debugfs_info_ops);
&ccp_debugfs_info_ops);
if (!debugfs_info)
goto err;
debugfs_stats = debugfs_create_file("stats", 0600, debugfs_create_file("stats", 0600, ccp->debugfs_instance, ccp,
ccp->debugfs_instance, ccp, &ccp_debugfs_stats_ops);
&ccp_debugfs_stats_ops);
if (!debugfs_stats)
goto err;
for (i = 0; i < ccp->cmd_q_count; i++) { for (i = 0; i < ccp->cmd_q_count; i++) {
cmd_q = &ccp->cmd_q[i]; cmd_q = &ccp->cmd_q[i];
@ -325,21 +312,12 @@ void ccp5_debugfs_setup(struct ccp_device *ccp)
debugfs_q_instance = debugfs_q_instance =
debugfs_create_dir(name, ccp->debugfs_instance); debugfs_create_dir(name, ccp->debugfs_instance);
if (!debugfs_q_instance)
goto err;
debugfs_q_stats = debugfs_create_file("stats", 0600, debugfs_q_instance, cmd_q,
debugfs_create_file("stats", 0600, &ccp_debugfs_queue_ops);
debugfs_q_instance, cmd_q,
&ccp_debugfs_queue_ops);
if (!debugfs_q_stats)
goto err;
} }
return; return;
err:
debugfs_remove_recursive(ccp->debugfs_instance);
} }
void ccp5_debugfs_destroy(void) void ccp5_debugfs_destroy(void)

2
drivers/crypto/ccp/ccp-ops.c
View File

@ -1,7 +1,7 @@
/* /*
* AMD Cryptographic Coprocessor (CCP) driver * AMD Cryptographic Coprocessor (CCP) driver
* *
* Copyright (C) 2013,2017 Advanced Micro Devices, Inc. * Copyright (C) 2013,2018 Advanced Micro Devices, Inc.
* *
* Author: Tom Lendacky <thomas.lendacky@amd.com> * Author: Tom Lendacky <thomas.lendacky@amd.com>
* Author: Gary R Hook <gary.hook@amd.com> * Author: Gary R Hook <gary.hook@amd.com>

37
drivers/crypto/ccp/psp-dev.c
View File

@ -1,7 +1,7 @@
/* /*
* AMD Platform Security Processor (PSP) interface * AMD Platform Security Processor (PSP) interface
* *
* Copyright (C) 2016-2017 Advanced Micro Devices, Inc. * Copyright (C) 2016,2018 Advanced Micro Devices, Inc.
* *
* Author: Brijesh Singh <brijesh.singh@amd.com> * Author: Brijesh Singh <brijesh.singh@amd.com>
* *
@ -437,6 +437,7 @@ static int sev_get_api_version(void)
psp_master->api_major = status->api_major; psp_master->api_major = status->api_major;
psp_master->api_minor = status->api_minor; psp_master->api_minor = status->api_minor;
psp_master->build = status->build; psp_master->build = status->build;
psp_master->sev_state = status->state;
return 0; return 0;
} }
@ -857,15 +858,15 @@ static int sev_misc_init(struct psp_device *psp)
return 0; return 0;
} }
static int sev_init(struct psp_device *psp) static int psp_check_sev_support(struct psp_device *psp)
{ {
/* Check if device supports SEV feature */ /* Check if device supports SEV feature */
if (!(ioread32(psp->io_regs + psp->vdata->feature_reg) & 1)) { if (!(ioread32(psp->io_regs + psp->vdata->feature_reg) & 1)) {
dev_dbg(psp->dev, "device does not support SEV\n"); dev_dbg(psp->dev, "psp does not support SEV\n");
return 1; return -ENODEV;
} }
return sev_misc_init(psp); return 0;
} }
int psp_dev_init(struct sp_device *sp) int psp_dev_init(struct sp_device *sp)
@ -890,6 +891,10 @@ int psp_dev_init(struct sp_device *sp)
psp->io_regs = sp->io_map; psp->io_regs = sp->io_map;
ret = psp_check_sev_support(psp);
if (ret)
goto e_disable;
/* Disable and clear interrupts until ready */ /* Disable and clear interrupts until ready */
iowrite32(0, psp->io_regs + psp->vdata->inten_reg); iowrite32(0, psp->io_regs + psp->vdata->inten_reg);
iowrite32(-1, psp->io_regs + psp->vdata->intsts_reg); iowrite32(-1, psp->io_regs + psp->vdata->intsts_reg);
@ -901,7 +906,7 @@ int psp_dev_init(struct sp_device *sp)
goto e_err; goto e_err;
} }
ret = sev_init(psp); ret = sev_misc_init(psp);
if (ret) if (ret)
goto e_irq; goto e_irq;
@ -922,6 +927,11 @@ e_err:
dev_notice(dev, "psp initialization failed\n"); dev_notice(dev, "psp initialization failed\n");
return ret;
e_disable:
sp->psp_data = NULL;
return ret; return ret;
} }
@ -964,6 +974,21 @@ void psp_pci_init(void)
if (sev_get_api_version()) if (sev_get_api_version())
goto err; goto err;
/*
* If platform is not in UNINIT state then firmware upgrade and/or
* platform INIT command will fail. These command require UNINIT state.
*
* In a normal boot we should never run into case where the firmware
* is not in UNINIT state on boot. But in case of kexec boot, a reboot
* may not go through a typical shutdown sequence and may leave the
* firmware in INIT or WORKING state.
*/
if (psp_master->sev_state != SEV_STATE_UNINIT) {
sev_platform_shutdown(NULL);
psp_master->sev_state = SEV_STATE_UNINIT;
}
if (SEV_VERSION_GREATER_OR_EQUAL(0, 15) && if (SEV_VERSION_GREATER_OR_EQUAL(0, 15) &&
sev_update_firmware(psp_master->dev) == 0) sev_update_firmware(psp_master->dev) == 0)
sev_get_api_version(); sev_get_api_version();

2
drivers/crypto/ccp/psp-dev.h
View File

@ -1,7 +1,7 @@
/* /*
* AMD Platform Security Processor (PSP) interface driver * AMD Platform Security Processor (PSP) interface driver
* *
* Copyright (C) 2017 Advanced Micro Devices, Inc. * Copyright (C) 2017-2018 Advanced Micro Devices, Inc.
* *
* Author: Brijesh Singh <brijesh.singh@amd.com> * Author: Brijesh Singh <brijesh.singh@amd.com>
* *

2
drivers/crypto/ccp/sp-dev.c
View File

@ -1,7 +1,7 @@
/* /*
* AMD Secure Processor driver * AMD Secure Processor driver
* *
* Copyright (C) 2017 Advanced Micro Devices, Inc. * Copyright (C) 2017-2018 Advanced Micro Devices, Inc.
* *
* Author: Tom Lendacky <thomas.lendacky@amd.com> * Author: Tom Lendacky <thomas.lendacky@amd.com>
* Author: Gary R Hook <gary.hook@amd.com> * Author: Gary R Hook <gary.hook@amd.com>

2
drivers/crypto/ccp/sp-dev.h
View File

@ -1,7 +1,7 @@
/* /*
* AMD Secure Processor driver * AMD Secure Processor driver
* *
* Copyright (C) 2017 Advanced Micro Devices, Inc. * Copyright (C) 2017-2018 Advanced Micro Devices, Inc.
* *
* Author: Tom Lendacky <thomas.lendacky@amd.com> * Author: Tom Lendacky <thomas.lendacky@amd.com>
* Author: Gary R Hook <gary.hook@amd.com> * Author: Gary R Hook <gary.hook@amd.com>

Some files were not shown because too many files have changed in this diff Show More