linux_dsm_epyc7002/drivers/nvme/host
Israel Rukshin 38e1800275 nvme-rdma: Avoid preallocating big SGL for data
nvme_rdma_alloc_tagset() preallocates a big buffer for the IO SGL based
on SG_CHUNK_SIZE.

Modern DMA engines are often capable of dealing with very big segments so
the SG_CHUNK_SIZE is often too big. SG_CHUNK_SIZE results in a static 4KB
SGL allocation per command.

If a controller has lots of deep queues, preallocation for the sg list can
consume substantial amounts of memory. For nvme-rdma, nr_hw_queues can be
128 and each queue's depth 128. This means the resulting preallocation
for the data SGL is 128*128*4K = 64MB per controller.

Switch to runtime allocation for SGL for lists longer than 2 entries. This
is the approach used by NVMe PCI so it should be reasonable for NVMeOF as
well. Runtime SGL allocation has always been the case for the legacy I/O
path so this is nothing new.

The preallocated small SGL depends on SG_CHAIN so if the ARCH doesn't
support SG_CHAIN, use only runtime allocation for the SGL.

We didn't notice of a performance degradation, since for small IOs we'll
use the inline SG and for the bigger IOs the allocation of a bigger SGL
from slab is fast enough.

Suggested-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Max Gurtovoy <maxg@mellanox.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Israel Rukshin <israelr@mellanox.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
2019-11-27 02:13:45 +09:00
..
core.c for-5.5/drivers-post-20191122 2019-11-25 11:18:03 -08:00
fabrics.c nvme-fabrics: allow discovery subsystems accept a kato 2019-09-12 08:50:46 -07:00
fabrics.h nvme: Remove ADMIN_ONLY state 2019-10-14 23:21:44 +09:00
fault_inject.c nvme: enable to inject errors into admin commands 2019-06-21 11:15:50 +02:00
fc.c nvme: move common call to nvme_cleanup_cmd to core layer 2019-11-04 10:56:41 -07:00
hwmon.c nvme: hwmon: add quirk to avoid changing temperature threshold 2019-11-22 02:21:08 +09:00
Kconfig nvme: Add hardware monitoring support 2019-11-12 01:57:35 +09:00
lightnvm.c lightnvm: move metadata mapping to lower level driver 2019-08-06 08:20:10 -06:00
Makefile nvme: Add hardware monitoring support 2019-11-12 01:57:35 +09:00
multipath.c for-5.5/drivers-20191121 2019-11-25 11:15:41 -08:00
nvme.h nvme-rdma: Avoid preallocating big SGL for data 2019-11-27 02:13:45 +09:00
pci.c for-5.5/drivers-post-20191122 2019-11-25 11:18:03 -08:00
rdma.c nvme-rdma: Avoid preallocating big SGL for data 2019-11-27 02:13:45 +09:00
tcp.c for-5.5/drivers-20191121 2019-11-25 11:15:41 -08:00
trace.c nvme: trace: parse Get LBA Status command in detail 2019-08-29 12:55:01 -07:00
trace.h nvme-trace: print result and status in hex format 2019-06-21 11:12:37 +02:00