linux_dsm_epyc7002/arch/x86/crypto
Eric Biggers 8aeef492fe crypto: x86/sha-mb - decrease priority of multibuffer algorithms
With all the crypto modules enabled on x86, and with a CPU that supports
AVX-2 but not SHA-NI instructions (e.g. Haswell, Broadwell, Skylake),
the "multibuffer" implementations of SHA-1, SHA-256, and SHA-512 are the
highest priority.  However, these implementations only perform well when
many hash requests are being submitted concurrently, filling all 8 AVX-2
lanes.  Otherwise, they are incredibly slow, as they waste time waiting
for more requests to arrive before proceeding to execute each request.

For example, here are the speeds I see hashing 4096-byte buffers with a
single thread on a Haswell-based processor:

            generic            avx2          mb (multibuffer)
            -------            --------      ----------------
sha1        602 MB/s           997 MB/s      0.61 MB/s
sha256      228 MB/s           412 MB/s      0.61 MB/s
sha512      312 MB/s           559 MB/s      0.61 MB/s

So, the multibuffer implementation is 500 to 1000 times slower than the
other implementations.  Note that with smaller buffers or more update()s
per digest, the difference would be even greater.

I believe the vast majority of people are in the boat where the
multibuffer code is much slower, and only a small minority are doing the
highly parallel, hashing-intensive, latency-flexible workloads (maybe
IPsec on servers?) where the multibuffer code may be beneficial.  Yet,
people often aren't familiar with all the crypto config options and so
the multibuffer code may inadvertently be built into the kernel.

Also the multibuffer code apparently hasn't been very well tested,
seeing as it was sometimes computing the wrong SHA-256 digest.

So, let's make the multibuffer algorithms low priority.  Users who want
to use them can either request them explicitly by driver name, or use
NETLINK_CRYPTO (crypto_user) to increase their priority at runtime.

Signed-off-by: Eric Biggers <ebiggers@google.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2018-07-09 00:30:21 +08:00
..
sha1-mb crypto: x86/sha-mb - decrease priority of multibuffer algorithms 2018-07-09 00:30:21 +08:00
sha256-mb crypto: x86/sha-mb - decrease priority of multibuffer algorithms 2018-07-09 00:30:21 +08:00
sha512-mb crypto: x86/sha-mb - decrease priority of multibuffer algorithms 2018-07-09 00:30:21 +08:00
aegis128-aesni-asm.S crypto: x86 - Add optimized AEGIS implementations 2018-05-19 00:14:00 +08:00
aegis128-aesni-glue.c crypto: x86 - Add optimized AEGIS implementations 2018-05-19 00:14:00 +08:00
aegis128l-aesni-asm.S crypto: x86 - Add optimized AEGIS implementations 2018-05-19 00:14:00 +08:00
aegis128l-aesni-glue.c crypto: x86 - Add optimized AEGIS implementations 2018-05-19 00:14:00 +08:00
aegis256-aesni-asm.S crypto: x86 - Add optimized AEGIS implementations 2018-05-19 00:14:00 +08:00
aegis256-aesni-glue.c crypto: x86/aegis256 - Fix wrong key buffer size 2018-05-27 00:12:12 +08:00
aes_ctrby8_avx-x86_64.S crypto, x86: aesni - fix token pasting for clang 2017-03-24 22:02:55 +08:00
aes_glue.c crypto: prefix module autoloading with "crypto-" 2014-11-24 22:43:57 +08:00
aes-i586-asm_32.S
aes-x86_64-asm_64.S crypto: x86/aes - Don't use %rbp as temporary register 2017-05-23 12:52:05 +08:00
aesni-intel_asm.S crypto: aesni - Introduce scatter/gather asm function stubs 2018-02-22 22:16:51 +08:00
aesni-intel_avx-x86_64.S crypto: aesni - make AVX2 AES-GCM work with all valid auth_tag_len 2017-05-18 13:19:55 +08:00
aesni-intel_glue.c crypto: aesni - Update aesni-intel_glue to use scatter/gather 2018-02-22 22:16:52 +08:00
blowfish_glue.c crypto: x86/blowfish: convert to skcipher interface 2018-03-03 00:03:29 +08:00
blowfish-x86_64-asm_64.S crypto: x86/blowfish - Fix RBP usage 2017-09-20 17:42:31 +08:00
camellia_aesni_avx2_glue.c crypto: x86/camellia-aesni-avx, avx2 - convert to skcipher interface 2018-03-03 00:03:32 +08:00
camellia_aesni_avx_glue.c crypto: x86/camellia-aesni-avx, avx2 - convert to skcipher interface 2018-03-03 00:03:32 +08:00
camellia_glue.c crypto: x86/camellia - convert to skcipher interface 2018-03-03 00:03:32 +08:00
camellia-aesni-avx2-asm_64.S x86/retpoline/crypto: Convert crypto assembler indirect jumps 2018-01-12 00:14:29 +01:00
camellia-aesni-avx-asm_64.S x86/retpoline/crypto: Convert crypto assembler indirect jumps 2018-01-12 00:14:29 +01:00
camellia-x86_64-asm_64.S crypto: x86/camellia - Fix RBP usage 2017-09-20 17:42:31 +08:00
cast5_avx_glue.c crypto: x86/glue_helper - rename glue_skwalk_fpu_begin() 2018-03-03 00:03:35 +08:00
cast5-avx-x86_64-asm_64.S crypto: x86/cast5 - Fix RBP usage 2017-09-20 17:42:32 +08:00
cast6_avx_glue.c crypto: x86/cast6-avx - convert to skcipher interface 2018-03-03 00:03:28 +08:00
cast6-avx-x86_64-asm_64.S crypto: x86/cast6 - Fix RBP usage 2017-09-20 17:42:33 +08:00
chacha20_glue.c crypto: x86/chacha20 - Remove cra_alignmask 2017-11-29 17:33:33 +11:00
chacha20-avx2-x86_64.S crypto: x86/chacha20 - satisfy stack validation 2.0 2017-10-12 22:51:16 +08:00
chacha20-ssse3-x86_64.S crypto: x86/chacha20 - satisfy stack validation 2.0 2017-10-12 22:51:16 +08:00
crc32-pclmul_asm.S crypto: crc32-pclmul - remove useless relative addressing 2017-10-07 12:10:30 +08:00
crc32-pclmul_glue.c crypto: hash - annotate algorithms taking optional key 2018-01-12 23:03:35 +11:00
crc32c-intel_glue.c crypto: hash - annotate algorithms taking optional key 2018-01-12 23:03:35 +11:00
crc32c-pcl-intel-asm_64.S x86/retpoline/crypto: Convert crypto assembler indirect jumps 2018-01-12 00:14:29 +01:00
crct10dif-pcl-asm_64.S crypto: x86 - make constants readonly, allow linker to merge them 2017-01-23 22:50:29 +08:00
crct10dif-pclmul_glue.c x86/cpufeature: Carve out X86_FEATURE_* 2016-01-30 11:22:17 +01:00
des3_ede_glue.c crypto: x86/des3_ede - des3_ede_skciphers[] can be static 2018-03-09 22:45:53 +08:00
des3_ede-asm_64.S crypto: x86/des3_ede - Fix RBP usage 2017-09-20 17:42:34 +08:00
fpu.c crypto: aesni - Convert to skcipher 2016-11-28 21:23:20 +08:00
ghash-clmulni-intel_asm.S crypto: x86 - make constants readonly, allow linker to merge them 2017-01-23 22:50:29 +08:00
ghash-clmulni-intel_glue.c crypto: ghash-clmulni - fix spelling mistake: "acclerated" -> "accelerated" 2018-05-05 14:52:58 +08:00
glue_helper-asm-avx2.S
glue_helper-asm-avx.S
glue_helper.c crypto: x86/glue_helper - rename glue_skwalk_fpu_begin() 2018-03-03 00:03:35 +08:00
Makefile crypto: x86/salsa20 - remove x86 salsa20 implementations 2018-05-31 00:13:57 +08:00
morus640_glue.c crypto: morus - Mark MORUS SIMD glue as x86-specific 2018-05-31 00:13:41 +08:00
morus640-sse2-asm.S crypto: x86 - Add optimized MORUS implementations 2018-05-19 00:15:35 +08:00
morus640-sse2-glue.c crypto: x86 - Add optimized MORUS implementations 2018-05-19 00:15:35 +08:00
morus1280_glue.c crypto: morus - Mark MORUS SIMD glue as x86-specific 2018-05-31 00:13:41 +08:00
morus1280-avx2-asm.S crypto: x86 - Add optimized MORUS implementations 2018-05-19 00:15:35 +08:00
morus1280-avx2-glue.c crypto: x86 - Add optimized MORUS implementations 2018-05-19 00:15:35 +08:00
morus1280-sse2-asm.S crypto: x86 - Add optimized MORUS implementations 2018-05-19 00:15:35 +08:00
morus1280-sse2-glue.c crypto: x86 - Add optimized MORUS implementations 2018-05-19 00:15:35 +08:00
poly1305_glue.c crypto: poly1305 - remove ->setkey() method 2018-01-12 23:03:14 +11:00
poly1305-avx2-x86_64.S crypto: x86 - make constants readonly, allow linker to merge them 2017-01-23 22:50:29 +08:00
poly1305-sse2-x86_64.S crypto: x86 - make constants readonly, allow linker to merge them 2017-01-23 22:50:29 +08:00
serpent_avx2_glue.c crypto: x86/serpent-avx,avx2 - convert to skcipher interface 2018-03-03 00:03:22 +08:00
serpent_avx_glue.c crypto: x86/serpent-avx,avx2 - convert to skcipher interface 2018-03-03 00:03:22 +08:00
serpent_sse2_glue.c crypto: x86/serpent-sse2 - convert to skcipher interface 2018-03-03 00:03:20 +08:00
serpent-avx2-asm_64.S crypto: x86 - make constants readonly, allow linker to merge them 2017-01-23 22:50:29 +08:00
serpent-avx-x86_64-asm_64.S crypto: x86 - make constants readonly, allow linker to merge them 2017-01-23 22:50:29 +08:00
serpent-sse2-i586-asm_32.S
serpent-sse2-x86_64-asm_64.S
sha1_avx2_x86_64_asm.S crypto: x86/sha1-avx2 - Fix RBP usage 2017-09-20 17:42:34 +08:00
sha1_ni_asm.S crypto: x86 - make constants readonly, allow linker to merge them 2017-01-23 22:50:29 +08:00
sha1_ssse3_asm.S crypto: x86/sha1-ssse3 - Fix RBP usage 2017-09-20 17:42:35 +08:00
sha1_ssse3_glue.c crypto: x86/sha1 - Fix reads beyond the number of blocks passed 2017-08-09 20:01:37 +08:00
sha256_ni_asm.S crypto: x86 - make constants readonly, allow linker to merge them 2017-01-23 22:50:29 +08:00
sha256_ssse3_glue.c crypto: sha-ssse3 - add MODULE_ALIAS 2016-05-31 16:41:46 +08:00
sha256-avx2-asm.S crypto: x86/sha256-avx2 - Fix RBP usage 2017-09-20 17:42:36 +08:00
sha256-avx-asm.S crypto: x86/sha256-avx - Fix RBP usage 2017-09-20 17:42:36 +08:00
sha256-ssse3-asm.S crypto: x86/sha256-ssse3 - Fix RBP usage 2017-09-20 17:42:37 +08:00
sha512_ssse3_glue.c crypto: sha-ssse3 - add MODULE_ALIAS 2016-05-31 16:41:46 +08:00
sha512-avx2-asm.S crypto: sha512-avx2 - Fix RBP usage 2017-09-20 17:42:37 +08:00
sha512-avx-asm.S crypto: x86 - make constants readonly, allow linker to merge them 2017-01-23 22:50:29 +08:00
sha512-ssse3-asm.S crypto: x86 - make constants readonly, allow linker to merge them 2017-01-23 22:50:29 +08:00
twofish_avx_glue.c crypto: x86/twofish-avx - convert to skcipher interface 2018-03-03 00:03:25 +08:00
twofish_glue_3way.c crypto: x86/twofish-3way - convert to skcipher interface 2018-03-03 00:03:24 +08:00
twofish_glue.c crypto: prefix module autoloading with "crypto-" 2014-11-24 22:43:57 +08:00
twofish-avx-x86_64-asm_64.S crypto: x86/twofish - Fix RBP usage 2017-09-20 17:42:38 +08:00
twofish-i586-asm_32.S
twofish-x86_64-asm_64-3way.S crypto: x86/twofish-3way - Fix %rbp usage 2017-12-28 17:56:44 +11:00
twofish-x86_64-asm_64.S x86/asm: Replace "MOVQ $imm, %reg" with MOVL 2015-04-01 13:17:39 +02:00