Go to file
Ard Biesheuvel bf93113d46 crypto: x86/aes-ni-xts - use direct calls to and 4-way stride
[ Upstream commit 86ad60a65f29dd862a11c22bb4b5be28d6c5cef1 ]

The XTS asm helper arrangement is a bit odd: the 8-way stride helper
consists of back-to-back calls to the 4-way core transforms, which
are called indirectly, based on a boolean that indicates whether we
are performing encryption or decryption.

Given how costly indirect calls are on x86, let's switch to direct
calls, and given how the 8-way stride doesn't really add anything
substantial, use a 4-way stride instead, and make the asm core
routine deal with any multiple of 4 blocks. Since 512 byte sectors
or 4 KB blocks are the typical quantities XTS operates on, increase
the stride exported to the glue helper to 512 bytes as well.

As a result, the number of indirect calls is reduced from 3 per 64 bytes
of in/output to 1 per 512 bytes of in/output, which produces a 65% speedup
when operating on 1 KB blocks (measured on a Intel(R) Core(TM) i7-8650U CPU)

Fixes: 9697fa39ef ("x86/retpoline/crypto: Convert crypto assembler indirect jumps")
Tested-by: Eric Biggers <ebiggers@google.com> # x86_64
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2021-03-20 10:43:43 +01:00
arch crypto: x86/aes-ni-xts - use direct calls to and 4-way stride 2021-03-20 10:43:43 +01:00
block block: Discard page cache of zone reset target range 2021-03-17 17:06:27 +01:00
certs certs: Fix blacklist flag type confusion 2021-03-04 11:37:59 +01:00
crypto crypto: mips/poly1305 - enable for all MIPS processors 2021-03-17 17:06:10 +01:00
Documentation KVM: arm64: Reject VM creation when the default IPA size is unsupported 2021-03-17 17:06:36 +01:00
drivers RDMA/umem: Use ib_dma_max_seg_size instead of dma_get_max_seg_size 2021-03-17 17:06:37 +01:00
fs binfmt_misc: fix possible deadlock in bm_register_write 2021-03-17 17:06:35 +01:00
include x86/entry: Move nmi entry/exit into common code 2021-03-17 17:06:36 +01:00
init kgdb: fix to kill breakpoints on initmem after boot 2021-03-04 11:38:46 +01:00
ipc ipc: adjust proc_ipc_sem_dointvec definition to match prototype 2020-09-05 12:14:29 -07:00
kernel x86/entry: Move nmi entry/exit into common code 2021-03-17 17:06:36 +01:00
lib kasan: fix memory corruption in kasan_bitops_tags test 2021-03-17 17:06:25 +01:00
LICENSES LICENSES/deprecated: add Zlib license text 2020-09-16 14:33:49 +02:00
mm mm/page_alloc.c: refactor initialization of struct page for holes in memory layout 2021-03-17 17:06:37 +01:00
net SUNRPC: Set memalloc_nofs_save() for sync tasks 2021-03-17 17:06:33 +01:00
samples samples, bpf: Add missing munmap in xdpsock 2021-03-17 17:06:12 +01:00
scripts ftrace: Have recordmcount use w8 to read relp->r_info in arm64_is_fake_mcount 2021-03-09 11:11:14 +01:00
security Revert 95ebabde382c ("capabilities: Don't allow writing ambiguous v3 file capabilities") 2021-03-17 17:06:27 +01:00
sound ALSA: usb-audio: fix use after free in usb_audio_disconnect 2021-03-17 17:06:27 +01:00
tools perf report: Fix -F for branch & mem modes 2021-03-17 17:06:17 +01:00
usr Merge branch 'work.fdpic' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs 2020-08-07 13:29:39 -07:00
virt KVM: Use kvm_pfn_t for local PFN variable in hva_to_pfn_remapped() 2021-02-26 10:13:01 +01:00
.clang-format RDMA 5.10 pull request 2020-10-17 11:18:18 -07:00
.cocciconfig
.get_maintainer.ignore Opt out of scripts/get_maintainer.pl 2019-05-16 10:53:40 -07:00
.gitattributes .gitattributes: use 'dts' diff driver for dts files 2019-12-04 19:44:11 -08:00
.gitignore .gitignore: docs: ignore sphinx_*/ directories 2020-09-10 10:44:31 -06:00
.mailmap mailmap: add two more addresses of Uwe Kleine-König 2020-12-06 10:19:07 -08:00
COPYING COPYING: state that all contributions really are covered by this file 2020-02-10 13:32:20 -08:00
CREDITS MAINTAINERS: Move Jason Cooper to CREDITS 2020-11-30 10:20:34 +01:00
Kbuild kbuild: rename hostprogs-y/always to hostprogs/always-y 2020-02-04 01:53:07 +09:00
Kconfig kbuild: ensure full rebuild when the compiler is updated 2020-05-12 13:28:33 +09:00
MAINTAINERS Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net 2020-12-10 15:30:13 -08:00
Makefile Linux 5.10.24 2021-03-17 17:06:37 +01:00
README

Linux kernel
============

There are several guides for kernel developers and users. These guides can
be rendered in a number of formats, like HTML and PDF. Please read
Documentation/admin-guide/README.rst first.

In order to build the documentation, use ``make htmldocs`` or
``make pdfdocs``.  The formatted documentation can also be read online at:

    https://www.kernel.org/doc/html/latest/

There are various text files in the Documentation/ subdirectory,
several of them using the Restructured Text markup notation.

Please read the Documentation/process/changes.rst file, as it contains the
requirements for building and running the kernel, and information about
the problems which may result by upgrading your kernel.