linux_dsm_epyc7002/Documentation/filesystems
Kirill A. Shutemov 8c6e50b029 mm: introduce vm_ops->map_pages()
Here's new version of faultaround patchset.  It took a while to tune it
and collect performance data.

First patch adds new callback ->map_pages to vm_operations_struct.

->map_pages() is called when VM asks to map easy accessible pages.
Filesystem should find and map pages associated with offsets from
"pgoff" till "max_pgoff".  ->map_pages() is called with page table
locked and must not block.  If it's not possible to reach a page without
blocking, filesystem should skip it.  Filesystem should use do_set_pte()
to setup page table entry.  Pointer to entry associated with offset
"pgoff" is passed in "pte" field in vm_fault structure.  Pointers to
entries for other offsets should be calculated relative to "pte".

Currently VM use ->map_pages only on read page fault path.  We try to
map FAULT_AROUND_PAGES a time.  FAULT_AROUND_PAGES is 16 for now.
Performance data for different FAULT_AROUND_ORDER is below.

TODO:
 - implement ->map_pages() for shmem/tmpfs;
 - modify get_user_pages() to be able to use ->map_pages() and implement
   mmap(MAP_POPULATE|MAP_NONBLOCK) on top.

=========================================================================
Tested on 4-socket machine (120 threads) with 128GiB of RAM.

Few real-world workloads. The sweet spot for FAULT_AROUND_ORDER here is
somewhere between 3 and 5. Let's say 4 :)

Linux build (make -j60)
FAULT_AROUND_ORDER		Baseline	1		3		4		5		7		9
	minor-faults		283,301,572	247,151,987	212,215,789	204,772,882	199,568,944	194,703,779	193,381,485
	time, seconds		151.227629483	153.920996480	151.356125472	150.863792049	150.879207877	151.150764954	151.450962358
Linux rebuild (make -j60)
FAULT_AROUND_ORDER		Baseline	1		3		4		5		7		9
	minor-faults		5,396,854	4,148,444	2,855,286	2,577,282	2,361,957	2,169,573	2,112,643
	time, seconds		27.404543757	27.559725591	27.030057426	26.855045126	26.678618635	26.974523490	26.761320095
Git test suite (make -j60 test)
FAULT_AROUND_ORDER		Baseline	1		3		4		5		7		9
	minor-faults		129,591,823	99,200,751	66,106,718	57,606,410	51,510,808	45,776,813	44,085,515
	time, seconds		66.087215026	64.784546905	64.401156567	65.282708668	66.034016829	66.793780811	67.237810413

Two synthetic tests: access every word in file in sequential/random order.
It doesn't improve much after FAULT_AROUND_ORDER == 4.

Sequential access 16GiB file
FAULT_AROUND_ORDER		Baseline	1		3		4		5		7		9
 1 thread
	minor-faults		4,195,437	2,098,275	525,068		262,251		131,170		32,856		8,282
	time, seconds		7.250461742	6.461711074	5.493859139	5.488488147	5.707213983	5.898510832	5.109232856
 8 threads
	minor-faults		33,557,540	16,892,728	4,515,848	2,366,999	1,423,382	442,732		142,339
	time, seconds		16.649304881	9.312555263	6.612490639	6.394316732	6.669827501	6.75078944	6.371900528
 32 threads
	minor-faults		134,228,222	67,526,810	17,725,386	9,716,537	4,763,731	1,668,921	537,200
	time, seconds		49.164430543	29.712060103	12.938649729	10.175151004	11.840094583	9.594081325	9.928461797
 60 threads
	minor-faults		251,687,988	126,146,952	32,919,406	18,208,804	10,458,947	2,733,907	928,217
	time, seconds		86.260656897	49.626551828	22.335007632	17.608243696	16.523119035	16.339489186	16.326390902
 120 threads
	minor-faults		503,352,863	252,939,677	67,039,168	35,191,827	19,170,091	4,688,357	1,471,862
	time, seconds		124.589206333	79.757867787	39.508707872	32.167281632	29.972989292	28.729834575	28.042251622
Random access 1GiB file
 1 thread
	minor-faults		262,636		132,743		34,369		17,299		8,527		3,451		1,222
	time, seconds		15.351890914	16.613802482	16.569227308	15.179220992	16.557356122	16.578247824	15.365266994
 8 threads
	minor-faults		2,098,948	1,061,871	273,690		154,501		87,110		25,663		7,384
	time, seconds		15.040026343	15.096933500	14.474757288	14.289129964	14.411537468	14.296316837	14.395635804
 32 threads
	minor-faults		8,390,734	4,231,023	1,054,432	528,847		269,242		97,746		26,881
	time, seconds		20.430433109	21.585235358	22.115062928	14.872878951	14.880856305	14.883370649	14.821261690
 60 threads
	minor-faults		15,733,258	7,892,809	1,973,393	988,266		594,789		164,994		51,691
	time, seconds		26.577302548	25.692397770	18.728863715	20.153026398	21.619101933	17.745086260	17.613215273
 120 threads
	minor-faults		31,471,111	15,816,616	3,959,209	1,978,685	1,008,299	264,635		96,010
	time, seconds		41.835322703	40.459786095	36.085306105	35.313894834	35.814445675	36.552633793	34.289210594

Touch only one page in page table in 16GiB file
FAULT_AROUND_ORDER		Baseline	1		3		4		5		7		9
 1 thread
	minor-faults		8,372		8,324		8,270		8,260		8,249		8,239		8,237
	time, seconds		0.039892712	0.045369149	0.051846126	0.063681685	0.079095975	0.17652406	0.541213386
 8 threads
	minor-faults		65,731		65,681		65,628		65,620		65,608		65,599		65,596
	time, seconds		0.124159196	0.488600638	0.156854426	0.191901957	0.242631486	0.543569456	1.677303984
 32 threads
	minor-faults		262,388		262,341		262,285		262,276		262,266		262,257		263,183
	time, seconds		0.452421421	0.488600638	0.565020946	0.648229739	0.789850823	1.651584361	5.000361559
 60 threads
	minor-faults		491,822		491,792		491,723		491,711		491,701		491,691		491,825
	time, seconds		0.763288616	0.869620515	0.980727360	1.161732354	1.466915814	3.04041448	9.308612938
 120 threads
	minor-faults		983,466		983,655		983,366		983,372		983,363		984,083		984,164
	time, seconds		1.595846553	1.667902182	2.008959376	2.425380942	2.941368804	5.977807890	18.401846125

This patch (of 2):

Introduce new vm_ops callback ->map_pages() and uses it for mapping easy
accessible pages around fault address.

On read page fault, if filesystem provides ->map_pages(), we try to map up
to FAULT_AROUND_PAGES pages around page fault address in hope to reduce
number of minor page faults.

We call ->map_pages first and use ->fault() as fallback if page by the
offset is not ready to be mapped (cold page cache or something).

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Rik van Riel <riel@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Matthew Wilcox <matthew.r.wilcox@intel.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Alexander Viro <viro@zeniv.linux.org.uk>
Cc: Dave Chinner <david@fromorbit.com>
Cc: Ning Qu <quning@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-04-07 16:35:52 -07:00
..
caching FS-Cache: Provide the ability to enable/disable cookies 2013-09-27 18:40:25 +01:00
cifs cifs: update cifs.txt and remove some outdated infos 2013-09-13 16:29:58 -05:00
configfs configfs: convert to umode_t 2012-01-03 22:54:57 -05:00
nfs Documentation/: update 00-INDEX files 2014-02-10 16:01:40 -08:00
pohmelfs Documentation: Fix multiple typo in Documentation 2012-03-07 16:08:24 +01:00
9p.txt 9p: update documentation 2014-01-24 10:55:21 -06:00
00-INDEX Documentation/: update 00-INDEX files 2014-02-10 16:01:40 -08:00
adfs.txt adfs: add hexadecimal filetype suffix option 2011-03-22 17:44:17 -07:00
affs.txt Documentation: update broken web addresses. 2010-08-04 15:21:40 +02:00
afs.txt AFS: Documentation updates 2009-08-19 10:40:13 -07:00
autofs4-mount-control.txt doc: fix double words 2014-03-21 13:16:58 +01:00
automount-support.txt VFS: Unexport do_kern_mount() and clean up simple_pin_fs() 2006-06-09 09:34:16 -04:00
befs.txt Documentation: befs.txt: no maintainer, orphaned 2011-08-13 18:34:03 -07:00
bfs.txt remove mention of CONFIG_KMOD from documentation 2008-07-22 19:24:29 +10:00
btrfs.txt btrfs: Add treelog mount option. 2014-01-28 13:20:20 -08:00
ceph.txt ceph: enable/disable dentry complete flags via mount option 2012-01-12 11:00:40 -08:00
coda.txt
cramfs.txt
debugfs.txt debugfs: more tightly restrict default mount mode 2012-08-27 13:42:02 -07:00
devpts.txt Document usage of multiple-instances of devpts 2009-01-02 10:19:36 -08:00
directory-locking vfs: take i_mutex on renamed file 2013-11-09 00:16:40 -05:00
dlmfs.txt Documentation/: it's -> its where appropriate 2010-04-23 02:09:52 +02:00
dnotify_test.c Documentation/fs/: split txt and source files 2010-03-12 15:52:35 -08:00
dnotify.txt Documentation/fs/: split txt and source files 2010-03-12 15:52:35 -08:00
ecryptfs.txt eCryptfs: Move ecryptfs docs into Documentation/filesystems/ 2007-07-17 10:23:08 -07:00
efivarfs.txt efivarfs: Add documentation for the EFI variable filesystem 2012-10-30 10:39:18 +00:00
exofs.txt exofs: Add option to mount by osdname 2011-03-15 15:02:51 +02:00
ext2.txt Doc fix: ext2 can only have 32,000 subdirs, not 32,768 2009-06-18 13:03:44 -07:00
ext3.txt ext3: allow specifying external journal by pathname mount option 2013-07-31 22:11:15 +02:00
ext4.txt Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial 2013-09-06 09:36:28 -07:00
f2fs.txt f2fs: update documents and a MAINTAINERS entry 2014-01-09 21:00:06 +09:00
fiemap.txt Documentation/: it's -> its where appropriate 2010-04-23 02:09:52 +02:00
files.txt Wrap accesses to the fd_sets in struct fdtable 2012-02-19 10:30:52 -08:00
fuse.txt Documentation/: it's -> its where appropriate 2010-04-23 02:09:52 +02:00
gfs2-glocks.txt GFS2: Update glock doc to add new stats info 2012-05-10 12:41:40 +01:00
gfs2-uevents.txt Documentation: Fix multiple typo in Documentation 2012-03-07 16:08:24 +01:00
gfs2.txt GFS2: Update main gfs2 doc 2012-05-10 11:45:31 +01:00
hfs.txt Documentation: HFS is orphaned 2011-11-04 12:01:48 -07:00
hfsplus.txt Documentation: update URL to hfsplus Technote 1150 2014-02-20 14:48:51 +01:00
hpfs.txt Documentation/: it's -> its where appropriate 2010-04-23 02:09:52 +02:00
inotify.txt Documentation: fix inotify source file paths 2011-11-04 12:01:47 -07:00
isofs.txt Documentation: update broken web addresses. 2010-08-04 15:21:40 +02:00
jfs.txt doc: fix misspellings with 'codespell' tool 2013-05-28 12:02:12 +02:00
Locking mm: introduce vm_ops->map_pages() 2014-04-07 16:35:52 -07:00
locks.txt doc: fix broken references 2011-09-27 18:08:04 +02:00
logfs.txt fix "seperate" typos in comments 2010-05-10 11:56:30 +02:00
Makefile Documentation/fs/: split txt and source files 2010-03-12 15:52:35 -08:00
mandatory-locking.txt locks: add warning about mandatory locking races 2007-10-09 18:32:45 -04:00
ncpfs.txt ncpfs: remove dead URL from documentation 2009-09-23 07:39:42 -07:00
nilfs2.txt nilfs2: update project's web site in nilfs2.txt 2014-04-03 16:21:26 -07:00
ntfs.txt Documentation/filesystems/ntfs.txt: remove changelog reference 2014-04-03 16:21:27 -07:00
ocfs2.txt ocfs2: clean up mount option about atime in ocfs2.txt 2011-05-23 23:37:12 -07:00
omfs.txt omfs: add filesystem documentation 2008-07-26 12:00:05 -07:00
path-lookup.txt Fix common misspellings 2011-03-31 11:26:23 -03:00
porting mm + fs: store shadow entries in page cache 2014-04-03 16:21:01 -07:00
proc.txt mm, oom: base root bonus on current usage 2014-01-30 16:56:56 -08:00
qnx6.txt doc: filesystems : Fix typo in Documentations/filesystems 2013-08-20 13:45:56 +02:00
quota.txt quota: documentation for sending "below quota" messages via netlink and tiny doc update 2008-08-12 16:07:27 -07:00
ramfs-rootfs-initramfs.txt initmpfs: use initramfs if rootfstype= or root= specified 2013-09-11 15:59:38 -07:00
relay.txt doc: Fix typo in doucmentations 2013-07-25 12:34:15 +02:00
romfs.txt Documentation/filesystems/romfs.txt: fixing link to genromfs 2011-02-17 22:04:46 +01:00
seq_file.txt seq_file: use proc_create() in documentation 2009-12-16 07:20:07 -08:00
sharedsubtree.txt Documentation: Fix trivial typo in filesystems/sharedsubtree.txt 2010-10-25 21:18:21 -04:00
spufs.txt Fix typos in /Documentation : 'U-Z' 2006-11-30 04:58:40 +01:00
squashfs.txt Squashfs: Update documentation to include xattrs 2011-12-30 01:20:24 +00:00
sysfs-pci.txt PCI: Allow read/write access to sysfs I/O port resources 2010-07-30 09:32:08 -07:00
sysfs-tagging.txt doc: Fix typo in doucmentations 2013-07-25 12:34:15 +02:00
sysfs.txt Documentation/filesystems/sysfs.txt: fix device_attribute declaration 2014-01-23 16:37:00 -08:00
sysv-fs.txt [PATCH] fs/sysv/: doc cleanup 2006-12-07 08:39:44 -08:00
tmpfs.txt mempolicy: document cpuset interaction with tmpfs mpol mount option 2010-05-25 08:06:57 -07:00
ubifs.txt UBIFS: switch self-check knobs to debugfs 2011-07-04 10:54:28 +03:00
udf.txt udf: implement mode and dmode mounting options 2009-04-02 12:29:50 +02:00
ufs.txt [PATCH] ufs2 write: mount as rw 2007-02-12 09:48:40 -08:00
vfat.txt Documentation/filesystems/vfat.txt: fix directory entry example 2013-11-13 12:09:32 +09:00
vfs.txt vfs: add renameat2 syscall 2014-04-01 17:08:42 +02:00
xfs-delayed-logging-design.txt Fix common misspellings 2011-03-31 11:26:23 -03:00
xfs-self-describing-metadata.txt xfs: add metadata CRC documentation 2013-04-27 13:27:43 -05:00
xfs.txt doc: filesystems : Fix typo in Documentations/filesystems 2013-08-20 13:45:56 +02:00
xip.txt DOC: update xip method info 2008-11-12 17:17:17 -08:00