linux_dsm_epyc7002/include/asm-generic
Minchan Kim 99baac21e4 mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem
Nadav reported parallel MADV_DONTNEED on same range has a stale TLB
problem and Mel fixed it[1] and found same problem on MADV_FREE[2].

Quote from Mel Gorman:
 "The race in question is CPU 0 running madv_free and updating some PTEs
  while CPU 1 is also running madv_free and looking at the same PTEs.
  CPU 1 may have writable TLB entries for a page but fail the pte_dirty
  check (because CPU 0 has updated it already) and potentially fail to
  flush.

  Hence, when madv_free on CPU 1 returns, there are still potentially
  writable TLB entries and the underlying PTE is still present so that a
  subsequent write does not necessarily propagate the dirty bit to the
  underlying PTE any more. Reclaim at some unknown time at the future
  may then see that the PTE is still clean and discard the page even
  though a write has happened in the meantime. I think this is possible
  but I could have missed some protection in madv_free that prevents it
  happening."

This patch aims for solving both problems all at once and is ready for
other problem with KSM, MADV_FREE and soft-dirty story[3].

TLB batch API(tlb_[gather|finish]_mmu] uses [inc|dec]_tlb_flush_pending
and mmu_tlb_flush_pending so that when tlb_finish_mmu is called, we can
catch there are parallel threads going on.  In that case, forcefully,
flush TLB to prevent for user to access memory via stale TLB entry
although it fail to gather page table entry.

I confirmed this patch works with [4] test program Nadav gave so this
patch supersedes "mm: Always flush VMA ranges affected by zap_page_range
v2" in current mmotm.

NOTE:

This patch modifies arch-specific TLB gathering interface(x86, ia64,
s390, sh, um).  It seems most of architecture are straightforward but
s390 need to be careful because tlb_flush_mmu works only if
mm->context.flush_mm is set to non-zero which happens only a pte entry
really is cleared by ptep_get_and_clear and friends.  However, this
problem never changes the pte entries but need to flush to prevent
memory access from stale tlb.

[1] http://lkml.kernel.org/r/20170725101230.5v7gvnjmcnkzzql3@techsingularity.net
[2] http://lkml.kernel.org/r/20170725100722.2dxnmgypmwnrfawp@suse.de
[3] http://lkml.kernel.org/r/BD3A0EBE-ECF4-41D4-87FA-C755EA9AB6BD@gmail.com
[4] https://patchwork.kernel.org/patch/9861621/

[minchan@kernel.org: decrease tlb flush pending count in tlb_finish_mmu]
  Link: http://lkml.kernel.org/r/20170808080821.GA31730@bbox
Link: http://lkml.kernel.org/r/20170802000818.4760-7-namit@vmware.com
Signed-off-by: Minchan Kim <minchan@kernel.org>
Signed-off-by: Nadav Amit <namit@vmware.com>
Reported-by: Nadav Amit <namit@vmware.com>
Reported-by: Mel Gorman <mgorman@techsingularity.net>
Acked-by: Mel Gorman <mgorman@techsingularity.net>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Yoshinori Sato <ysato@users.sourceforge.jp>
Cc: Jeff Dike <jdike@addtoit.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Nadav Amit <nadav.amit@gmail.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2017-08-10 15:54:07 -07:00
..
bitops bitops: Do not default to __clear_bit() for __clear_bit_unlock() 2016-03-21 10:50:48 +01:00
4level-fixup.h asm-generic: introduce 5level-fixup.h 2017-03-09 11:48:47 -08:00
5level-fixup.h asm-generic: introduce 5level-fixup.h 2017-03-09 11:48:47 -08:00
asm-offsets.h asm-generic: Add common asm-offsets.h 2015-06-23 13:35:49 +09:00
asm-prototypes.h asm-prototypes: Clear any CPP defines before declaring the functions 2017-01-05 12:57:42 +01:00
atomic64.h locking/atomic: Implement atomic{,64,_long}_fetch_{add,sub,and,andnot,or,xor}{,_relaxed,_acquire,_release}() 2016-06-16 10:48:32 +02:00
atomic-long.h locking/atomic: Introduce inc/dec variants for the atomic_fetch_$op() API 2016-07-07 09:16:20 +02:00
atomic.h openrisc: add optimized atomic operations 2017-02-25 04:14:06 +09:00
audit_change_attr.h audit: Modify a set of system calls in audit class definitions 2014-01-17 17:01:46 -05:00
audit_dir_write.h audit: support the "standard" <asm-generic/unistd.h> 2011-05-04 14:41:28 -04:00
audit_read.h audit: support the "standard" <asm-generic/unistd.h> 2011-05-04 14:41:28 -04:00
audit_signal.h
audit_write.h audit: Modify a set of system calls in audit class definitions 2014-01-17 17:01:46 -05:00
barrier.h locking/spinlock, arch: Update and fix spin_unlock_wait() implementations 2016-06-14 11:55:15 +02:00
bitops.h arch: Prepare for smp_mb__{before,after}_atomic() 2014-04-18 11:40:30 +02:00
bitsperlong.h UAPI: (Scripted) Disintegrate include/asm-generic 2012-10-04 18:20:15 +01:00
bug.h asm-generic/bug.h: declare struct pt_regs; before function prototype 2017-07-10 16:32:34 -07:00
bugs.h
cache.h
cacheflush.h asm-generic/cacheflush.h: flush icache when copying to user pages 2011-05-25 08:39:37 -07:00
checksum.h ipv4: Update parameters for csum_tcpudp_magic to their original types 2016-03-13 23:55:13 -04:00
clkdev.h asm-generic: COMMON_CLK defines __clk_{get,put} 2014-09-25 18:00:45 -07:00
cmpxchg-local.h LLVMLinux: Remove warning about returning an uninitialized variable 2014-04-09 13:44:35 -07:00
cmpxchg.h asm-generic: cmpxchg: avoid warnings from macro-ized cmpxchg() implementations 2015-10-15 00:21:13 +02:00
current.h
delay.h asm-generic: delay.h fix udelay and ndelay for 8 bit args 2011-07-22 18:45:33 +02:00
device.h
div64.h __div64_32(): make it overridable at compile time 2015-11-16 14:42:12 -05:00
dma-contiguous.h asm-generic: Add dma-contiguous.h 2014-09-22 13:35:51 +02:00
dma.h
early_ioremap.h Merge branch 'akpm' (patches from Andrew) 2015-09-08 17:52:23 -07:00
emergency-restart.h
exec.h Split arch_align_stack() out from asm-generic/system.h 2012-03-28 18:30:03 +01:00
export.h modversions: treat symbol CRCs as 32 bit quantities 2017-02-03 08:28:25 -08:00
extable.h add asm-generic/extable.h 2017-03-27 20:27:28 -04:00
fb.h
fixmap.h asm-generic: Fix local variable shadow in __set_fixmap_offset 2016-02-16 15:10:44 +00:00
ftrace.h
futex.h asm-generic/futex: Re-enable preemption in futex_atomic_cmpxchg_inatomic() 2016-04-21 11:06:09 +02:00
getorder.h bitops: Add missing parentheses to new get_order macro 2012-02-24 10:39:27 -08:00
gpio.h gpio: allow setting ARCH_NR_GPIOS from Kconfig 2016-02-19 00:22:54 +01:00
hardirq.h
hugetlb.h mm/hugetlb: allow architectures to override huge_pte_clear() 2017-07-06 16:24:34 -07:00
hw_irq.h
ide_iops.h
int-ll64.h UAPI: (Scripted) Disintegrate include/asm-generic 2012-10-04 18:20:15 +01:00
io.h asm-generic/io.h: add io{read,write}64 accessors 2016-05-31 16:41:50 +08:00
ioctl.h include/asm-generic/ioctl.h: fix _IOC_TYPECHECK sparse error 2014-06-06 16:08:13 -07:00
iomap.h asm-generic/io.h: add io{read,write}64 accessors 2016-05-31 16:41:50 +08:00
irq_regs.h
irq_work.h irq_work: Introduce arch_irq_work_has_interrupt() 2014-09-13 18:38:07 +02:00
irq.h
irqflags.h
kdebug.h
kmap_types.h asm-generic: remove km_type definitions 2012-07-24 15:27:30 +08:00
kprobes.h kprobes: move kprobe declarations to asm-generic/kprobes.h 2017-02-27 18:43:45 -08:00
kvm_para.h KVM: add kvm_para_available to asm-generic/kvm_para.h 2013-06-05 13:21:29 +03:00
linkage.h
local64.h atomic: use <linux/atomic.h> 2011-07-26 16:49:47 -07:00
local.h atomic: use <linux/atomic.h> 2011-07-26 16:49:47 -07:00
mcs_spinlock.h locking/mcs: Allow architecture specific asm files to be used for contended case 2014-02-09 21:18:52 +01:00
memory_model.h mm: add PHYS_PFN, use it in __phys_to_pfn() 2016-01-14 16:00:49 -08:00
mm_hooks.h mm/gup: Drop the arch_pte_access_permitted() MMU callback 2017-03-18 09:48:01 +01:00
mm-arch-hooks.h mm: clean up per architecture MM hook header files 2015-07-17 16:39:53 -07:00
mmu_context.h asm-generic: Remove asm-generic arch_bprm_mm_init() 2014-11-22 21:52:08 +01:00
mmu.h asm-generic/mmu.h: Add support for FDPIC 2012-12-09 23:14:14 +01:00
module.h Make most arch asm/module.h files use asm-generic/module.h 2012-09-28 14:31:03 +09:30
msi.h asm-generic: Add msi.h 2014-11-23 13:01:47 +01:00
page.h asm-generic: page.h: Remove useless get_user_page and free_user_page 2016-02-26 15:24:55 +01:00
param.h UAPI: (Scripted) Disintegrate include/asm-generic 2012-10-04 18:20:15 +01:00
parport.h include: remove __dev* attributes. 2013-01-03 15:57:16 -08:00
pci_iomap.h PCI: Add pci_iomap_wc() variants 2015-08-25 09:59:45 +02:00
pci.h PCI: Remove unused pcibios_select_root() (again) 2015-06-08 07:56:21 -05:00
percpu.h percpu: use notrace variant of preempt_disable/preempt_enable 2016-11-08 10:29:44 +01:00
pgalloc.h
pgtable-nop4d-hack.h asm-generic: introduce __ARCH_USE_5LEVEL_HACK 2017-03-09 11:48:47 -08:00
pgtable-nop4d.h asm-generic: introduce <asm-generic/pgtable-nop4d.h> 2017-03-09 11:48:47 -08:00
pgtable-nopmd.h
pgtable-nopud.h asm-generic: introduce <asm-generic/pgtable-nop4d.h> 2017-03-09 11:48:47 -08:00
pgtable.h mm/gup: Move permission checks into helpers 2017-03-18 09:48:01 +01:00
preempt.h sched/preempt: Fix preempt_count manipulations 2016-05-17 12:24:21 +02:00
ptrace.h asm-generic/ptrace.h: start a common low level ptrace helper 2011-05-26 17:12:36 -07:00
qrwlock_types.h locking/qrwlock: Rename ->lock to ->wait_lock 2015-09-18 09:27:29 +02:00
qrwlock.h locking/qrwlock: Fix write unlock bug on big endian systems 2016-08-10 14:13:27 +02:00
qspinlock_types.h locking/qspinlock: Move __ARCH_SPIN_LOCK_UNLOCKED to qspinlock_types.h 2016-02-29 10:02:43 +01:00
qspinlock.h locking/qspinlock: Use atomic_sub_return_release() in queued_spin_unlock() 2016-06-08 15:17:01 +02:00
resource.h asm-generic: remove _STK_LIM_MAX 2014-05-15 00:32:09 +01:00
rwsem.h locking/rwsem: Remove unnecessary atomic_long_t casts 2017-01-20 09:46:44 +01:00
seccomp.h seccomp: Get compat syscalls from asm-generic header 2016-05-13 14:02:00 +02:00
sections.h mm: fix section name for .data..ro_after_init 2017-03-31 17:13:30 -07:00
segment.h
serial.h
set_memory.h treewide: move set_memory_* functions away from cacheflush.h 2017-05-08 17:15:13 -07:00
signal.h unify default ptrace_signal_deliver 2012-11-29 00:01:23 -05:00
simd.h crypto: create generic version of ablk_helper 2013-09-24 06:02:24 +10:00
sizes.h ARM: 7430/1: sizes.h: move from asm-generic to <linux/sizes.h> 2012-06-28 17:14:34 +01:00
spinlock.h
statfs.h UAPI: (Scripted) Disintegrate include/asm-generic 2012-10-04 18:20:15 +01:00
string.h
switch_to.h Split the switch_to() wrapper out of asm-generic/system.h 2012-03-28 18:30:03 +01:00
syscall.h syscall.h: fix doc text for syscall_get_arch() 2014-09-23 16:20:00 -04:00
syscalls.h burying unused conditionals 2013-02-14 09:21:15 -05:00
termios-base.h Replace <asm/uaccess.h> with <linux/uaccess.h> globally 2016-12-24 11:46:01 -08:00
termios.h Replace <asm/uaccess.h> with <linux/uaccess.h> globally 2016-12-24 11:46:01 -08:00
timex.h
tlb.h mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem 2017-08-10 15:54:07 -07:00
tlbflush.h BUG: headers with BUG/BUG_ON etc. need linux/bug.h 2012-03-04 17:54:34 -05:00
topology.h
trace_clock.h tracing,x86: Add a TSC trace_clock 2012-11-13 15:48:27 -05:00
uaccess.h kill strlen_user() 2017-05-15 23:40:22 -04:00
unaligned.h asm-generic: allow generic unaligned access if the arch supports it 2014-05-08 10:22:23 +02:00
unistd.h We get rid of the general module prefix confusion with a binary config option, 2013-05-05 10:58:06 -07:00
user.h
vga.h
vmlinux.lds.h powerpc updates for 4.13 2017-07-07 13:55:45 -07:00
vtime.h include/asm-generic/vtime.h: avoid zero-length file 2013-09-30 14:31:02 -07:00
word-at-a-time.h Make asm/word-at-a-time.h available on all architectures 2015-07-08 16:41:55 -04:00
xor.h asm-generic: xor: mark static functions as __maybe_unused 2012-10-03 21:21:06 +02:00