AuxXxilium/linux_dsm_epyc7002
1
0
Fork 0
mirror of https://github.com/AuxXxilium/linux_dsm_epyc7002.git synced 2024-12-18 11:16:42 +07:00
Code Issues Actions Packages Projects Releases Wiki Activity
be4c2d4723
linux_dsm_epyc7002/fs/nfs
History
luanshi be4c2d4723 NFS: readdirplus optimization by cache mechanism 
When listing very large directories via NFS, clients may take a long
time to complete. There are about three factors involved:

First of all, ls and practically every other method of listing a
directory including python os.listdir and find rely on libc readdir().
However readdir() only reads 32K of directory entries at a time, which
means that if you have a lot of files in the same directory, it is going
to take an insanely long time to read all the directory entries.

Secondly, libc readdir() reads 32K of directory entries at a time, in
kernel space 32K buffer split into 8 pages. One NFS readdirplus rpc will
be called for one page, which introduces many readdirplus rpc calls.

Lastly, one NFS readdirplus rpc asks for 32K data (filled by nfs_dentry)
to fill one page (filled by dentry), we found that nearly one third of
data was wasted.

To solve above problems, pagecache mechanism was introduced. One NFS
readdirplus rpc will ask for a large data (more than 32k), the data can
fill more than one page, the cached pages can be used for next readdir
call. This can reduce many readdirplus rpc calls and improve readdirplus
performance.

TESTING:
When listing very large directories(include 300 thousand files) via NFS

time ls -l /nfs_mount | wc -l

without the patch:
300001
real    1m53.524s
user    0m2.314s
sys     0m2.599s

with the patch:
300001
real    0m23.487s
user    0m2.305s
sys     0m2.558s

Improved performance: 79.6%
readdirplus rpc calls decrease: 85%

Signed-off-by: Liguang Zhang <zhangliguang@linux.alibaba.com>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
2019-02-20 17:33:55 -05:00
..
blocklayout NFS/NFSD/SUNRPC: replace generic creds with 'struct cred'. 2018-12-19 13:52:46 -05:00
filelayout pNFS: Don't allocate more pages than we need to fit a layoutget response 2018-09-30 15:35:16 -04:00
flexfilelayout NFS: Fix up documentation warnings 2019-02-20 15:14:21 -05:00
cache_lib.c NFS client updates for Linux 4.15 2017-11-17 14:18:00 -08:00
cache_lib.h NFS client updates for Linux 4.15 2017-11-17 14:18:00 -08:00
callback_proc.c NFSv4.2 copy do not allocate memory under the lock 2018-11-22 13:54:46 -05:00
callback_xdr.c NFS CB_OFFLOAD xdr 2018-08-09 12:56:38 -04:00
callback.c nfs: fixed broken compilation in nfs_callback_up_net() 2018-12-31 11:25:16 -05:00
callback.h NFS CB_OFFLOAD xdr 2018-08-09 12:56:38 -04:00
client.c NFS/SUNRPC: don't lookup machine credential until rpcauth_bindcred(). 2018-12-19 13:52:45 -05:00
delegation.c NFS: Fix up documentation warnings 2019-02-20 15:14:21 -05:00
delegation.h NFS/NFSD/SUNRPC: replace generic creds with 'struct cred'. 2018-12-19 13:52:46 -05:00
dir.c NFS: readdirplus optimization by cache mechanism 2019-02-20 17:33:55 -05:00
direct.c NFS: Pass error information to the pgio error cleanup routine 2019-02-20 15:14:20 -05:00
dns_resolve.c NFS: Fix up a typo in nfs_dns_ent_put 2018-10-29 16:58:04 -04:00
dns_resolve.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
export.c NFS: Pass the inode down to the getattr() callback 2018-06-04 12:07:07 -04:00
file.c NFS: Fix up documentation warnings 2019-02-20 15:14:21 -05:00
fscache-index.c vfs: change inode times to use struct timespec64 2018-06-05 16:57:31 -07:00
fscache.c vfs: change inode times to use struct timespec64 2018-06-05 16:57:31 -07:00
fscache.h fscache: Pass object size in rather than calling back for it 2018-04-06 14:05:14 +01:00
getroot.c Replace <asm/uaccess.h> with <linux/uaccess.h> globally 2016-12-24 11:46:01 -08:00
inode.c NFS: Fix up documentation warnings 2019-02-20 15:14:21 -05:00
internal.h NFS: readdirplus optimization by cache mechanism 2019-02-20 17:33:55 -05:00
io.c NFS: Fix up documentation warnings 2019-02-20 15:14:21 -05:00
iostat.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
Kconfig pnfs/blocklayout: require 64-bit sector_t 2017-08-11 14:10:13 -04:00
Makefile License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
mount_clnt.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
namespace.c NFS: Fix up documentation warnings 2019-02-20 15:14:21 -05:00
netns.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
nfs2super.c NFS: Convert v2 into a module 2012-07-30 19:06:41 -04:00
nfs2xdr.c vfs: change inode times to use struct timespec64 2018-06-05 16:57:31 -07:00
nfs3_fs.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
nfs3acl.c NFS: Mark expected switch fall-throughs 2018-08-08 16:50:02 -04:00
nfs3client.c NFS: Remove unused authflavour parameter from nfs_get_client() 2016-12-01 17:46:32 -05:00
nfs3proc.c NFS/NFSD/SUNRPC: replace generic creds with 'struct cred'. 2018-12-19 13:52:46 -05:00
nfs3super.c nfsv3: introduce nfs3_set_ds_client 2015-02-03 11:06:34 -08:00
nfs3xdr.c NFSv3: Improve NFSv3 performance when server returns no post-op attributes 2018-09-30 15:35:17 -04:00
nfs4_fs.h sunrpc: Add xprt after nfs4_test_session_trunk() 2019-01-02 12:05:19 -05:00
nfs4client.c NFS: drop useless LIST_HEAD 2019-02-20 17:33:55 -05:00
nfs4file.c NFSv4.2 fix unnecessary retry in nfs4_copy_file_range 2019-01-15 11:24:49 -05:00
nfs4getroot.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
nfs4idmap.c keys: Fix dependency loop between construction record and auth key 2019-02-15 14:12:09 -08:00
nfs4idmap.h NFS: Move nfs_idmap.h into fs/nfs/ 2015-04-23 15:16:14 -04:00
nfs4namespace.c NFS: Fix up documentation warnings 2019-02-20 15:14:21 -05:00
nfs4proc.c NFS: Fix sparse annotations for nfs_set_open_stateid_locked() 2019-02-20 15:14:21 -05:00
nfs4renewd.c NFS/NFSD/SUNRPC: replace generic creds with 'struct cred'. 2018-12-19 13:52:46 -05:00
nfs4session.c NFS: Fix up documentation warnings 2019-02-20 15:14:21 -05:00
nfs4session.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
nfs4state.c NFS: Fix up documentation warnings 2019-02-20 15:14:21 -05:00
nfs4super.c NFS: Move nfs_idmap.h into fs/nfs/ 2015-04-23 15:16:14 -04:00
nfs4sysctl.c nfs: Do not convert nfs_idmap_cache_timeout to jiffies 2018-01-18 15:10:47 -05:00
nfs4trace.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
nfs4trace.h NFS: Fix NFSv4 symbolic trace point output 2019-01-02 12:05:18 -05:00
nfs4xdr.c NFS: change sign of nfs_fh length 2018-10-23 12:22:21 -04:00
nfs42.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
nfs42proc.c NFSv4.2 copy do not allocate memory under the lock 2018-11-22 13:54:46 -05:00
nfs42xdr.c NFS add support for asynchronous COPY 2018-08-09 12:56:39 -04:00
nfs.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
nfsroot.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
nfstrace.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
nfstrace.h NFS client updates for Linux 4.16 2018-01-30 19:03:48 -08:00
pagelist.c NFS: Fix up documentation warnings 2019-02-20 15:14:21 -05:00
pnfs_dev.c NFS/NFSD/SUNRPC: replace generic creds with 'struct cred'. 2018-12-19 13:52:46 -05:00
pnfs_nfs.c NFS/NFSD/SUNRPC: replace generic creds with 'struct cred'. 2018-12-19 13:52:46 -05:00
pnfs.c NFS/NFSD/SUNRPC: replace generic creds with 'struct cred'. 2018-12-19 13:52:46 -05:00
pnfs.h NFS/NFSD/SUNRPC: replace generic creds with 'struct cred'. 2018-12-19 13:52:46 -05:00
proc.c NFS/NFSD/SUNRPC: replace generic creds with 'struct cred'. 2018-12-19 13:52:46 -05:00
read.c NFS: Pass error information to the pgio error cleanup routine 2019-02-20 15:14:20 -05:00
super.c fs/nfs: Fix nfs_parse_devname to not modify it's argument 2019-02-20 17:33:55 -05:00
symlink.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
sysctl.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
unlink.c NFS: Fix up documentation warnings 2019-02-20 15:14:21 -05:00
write.c NFS: Fix up documentation warnings 2019-02-20 15:14:21 -05:00