2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* Generic hugetlb support.
|
|
|
|
* (C) William Irwin, April 2004
|
|
|
|
*/
|
|
|
|
#include <linux/gfp.h>
|
|
|
|
#include <linux/list.h>
|
|
|
|
#include <linux/init.h>
|
|
|
|
#include <linux/module.h>
|
|
|
|
#include <linux/mm.h>
|
2008-10-16 02:50:22 +07:00
|
|
|
#include <linux/seq_file.h>
|
2005-04-17 05:20:36 +07:00
|
|
|
#include <linux/sysctl.h>
|
|
|
|
#include <linux/highmem.h>
|
mmu-notifiers: core
With KVM/GFP/XPMEM there isn't just the primary CPU MMU pointing to pages.
There are secondary MMUs (with secondary sptes and secondary tlbs) too.
sptes in the kvm case are shadow pagetables, but when I say spte in
mmu-notifier context, I mean "secondary pte". In GRU case there's no
actual secondary pte and there's only a secondary tlb because the GRU
secondary MMU has no knowledge about sptes and every secondary tlb miss
event in the MMU always generates a page fault that has to be resolved by
the CPU (this is not the case of KVM where the a secondary tlb miss will
walk sptes in hardware and it will refill the secondary tlb transparently
to software if the corresponding spte is present). The same way
zap_page_range has to invalidate the pte before freeing the page, the spte
(and secondary tlb) must also be invalidated before any page is freed and
reused.
Currently we take a page_count pin on every page mapped by sptes, but that
means the pages can't be swapped whenever they're mapped by any spte
because they're part of the guest working set. Furthermore a spte unmap
event can immediately lead to a page to be freed when the pin is released
(so requiring the same complex and relatively slow tlb_gather smp safe
logic we have in zap_page_range and that can be avoided completely if the
spte unmap event doesn't require an unpin of the page previously mapped in
the secondary MMU).
The mmu notifiers allow kvm/GRU/XPMEM to attach to the tsk->mm and know
when the VM is swapping or freeing or doing anything on the primary MMU so
that the secondary MMU code can drop sptes before the pages are freed,
avoiding all page pinning and allowing 100% reliable swapping of guest
physical address space. Furthermore it avoids the code that teardown the
mappings of the secondary MMU, to implement a logic like tlb_gather in
zap_page_range that would require many IPI to flush other cpu tlbs, for
each fixed number of spte unmapped.
To make an example: if what happens on the primary MMU is a protection
downgrade (from writeable to wrprotect) the secondary MMU mappings will be
invalidated, and the next secondary-mmu-page-fault will call
get_user_pages and trigger a do_wp_page through get_user_pages if it
called get_user_pages with write=1, and it'll re-establishing an updated
spte or secondary-tlb-mapping on the copied page. Or it will setup a
readonly spte or readonly tlb mapping if it's a guest-read, if it calls
get_user_pages with write=0. This is just an example.
This allows to map any page pointed by any pte (and in turn visible in the
primary CPU MMU), into a secondary MMU (be it a pure tlb like GRU, or an
full MMU with both sptes and secondary-tlb like the shadow-pagetable layer
with kvm), or a remote DMA in software like XPMEM (hence needing of
schedule in XPMEM code to send the invalidate to the remote node, while no
need to schedule in kvm/gru as it's an immediate event like invalidating
primary-mmu pte).
At least for KVM without this patch it's impossible to swap guests
reliably. And having this feature and removing the page pin allows
several other optimizations that simplify life considerably.
Dependencies:
1) mm_take_all_locks() to register the mmu notifier when the whole VM
isn't doing anything with "mm". This allows mmu notifier users to keep
track if the VM is in the middle of the invalidate_range_begin/end
critical section with an atomic counter incraese in range_begin and
decreased in range_end. No secondary MMU page fault is allowed to map
any spte or secondary tlb reference, while the VM is in the middle of
range_begin/end as any page returned by get_user_pages in that critical
section could later immediately be freed without any further
->invalidate_page notification (invalidate_range_begin/end works on
ranges and ->invalidate_page isn't called immediately before freeing
the page). To stop all page freeing and pagetable overwrites the
mmap_sem must be taken in write mode and all other anon_vma/i_mmap
locks must be taken too.
2) It'd be a waste to add branches in the VM if nobody could possibly
run KVM/GRU/XPMEM on the kernel, so mmu notifiers will only enabled if
CONFIG_KVM=m/y. In the current kernel kvm won't yet take advantage of
mmu notifiers, but this already allows to compile a KVM external module
against a kernel with mmu notifiers enabled and from the next pull from
kvm.git we'll start using them. And GRU/XPMEM will also be able to
continue the development by enabling KVM=m in their config, until they
submit all GRU/XPMEM GPLv2 code to the mainline kernel. Then they can
also enable MMU_NOTIFIERS in the same way KVM does it (even if KVM=n).
This guarantees nobody selects MMU_NOTIFIER=y if KVM and GRU and XPMEM
are all =n.
The mmu_notifier_register call can fail because mm_take_all_locks may be
interrupted by a signal and return -EINTR. Because mmu_notifier_reigster
is used when a driver startup, a failure can be gracefully handled. Here
an example of the change applied to kvm to register the mmu notifiers.
Usually when a driver startups other allocations are required anyway and
-ENOMEM failure paths exists already.
struct kvm *kvm_arch_create_vm(void)
{
struct kvm *kvm = kzalloc(sizeof(struct kvm), GFP_KERNEL);
+ int err;
if (!kvm)
return ERR_PTR(-ENOMEM);
INIT_LIST_HEAD(&kvm->arch.active_mmu_pages);
+ kvm->arch.mmu_notifier.ops = &kvm_mmu_notifier_ops;
+ err = mmu_notifier_register(&kvm->arch.mmu_notifier, current->mm);
+ if (err) {
+ kfree(kvm);
+ return ERR_PTR(err);
+ }
+
return kvm;
}
mmu_notifier_unregister returns void and it's reliable.
The patch also adds a few needed but missing includes that would prevent
kernel to compile after these changes on non-x86 archs (x86 didn't need
them by luck).
[akpm@linux-foundation.org: coding-style fixes]
[akpm@linux-foundation.org: fix mm/filemap_xip.c build]
[akpm@linux-foundation.org: fix mm/mmu_notifier.c build]
Signed-off-by: Andrea Arcangeli <andrea@qumranet.com>
Signed-off-by: Nick Piggin <npiggin@suse.de>
Signed-off-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Jack Steiner <steiner@sgi.com>
Cc: Robin Holt <holt@sgi.com>
Cc: Nick Piggin <npiggin@suse.de>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Kanoj Sarcar <kanojsarcar@yahoo.com>
Cc: Roland Dreier <rdreier@cisco.com>
Cc: Steve Wise <swise@opengridcomputing.com>
Cc: Avi Kivity <avi@qumranet.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Anthony Liguori <aliguori@us.ibm.com>
Cc: Chris Wright <chrisw@redhat.com>
Cc: Marcelo Tosatti <marcelo@kvack.org>
Cc: Eric Dumazet <dada1@cosmosbay.com>
Cc: "Paul E. McKenney" <paulmck@us.ibm.com>
Cc: Izik Eidus <izike@qumranet.com>
Cc: Anthony Liguori <aliguori@us.ibm.com>
Cc: Rik van Riel <riel@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-29 05:46:29 +07:00
|
|
|
#include <linux/mmu_notifier.h>
|
2005-04-17 05:20:36 +07:00
|
|
|
#include <linux/nodemask.h>
|
2005-06-22 07:14:44 +07:00
|
|
|
#include <linux/pagemap.h>
|
2006-01-06 15:10:46 +07:00
|
|
|
#include <linux/mempolicy.h>
|
2006-01-08 16:00:57 +07:00
|
|
|
#include <linux/cpuset.h>
|
[PATCH] hugepage: serialize hugepage allocation and instantiation
Currently, no lock or mutex is held between allocating a hugepage and
inserting it into the pagetables / page cache. When we do go to insert the
page into pagetables or page cache, we recheck and may free the newly
allocated hugepage. However, since the number of hugepages in the system
is strictly limited, and it's usualy to want to use all of them, this can
still lead to spurious allocation failures.
For example, suppose two processes are both mapping (MAP_SHARED) the same
hugepage file, large enough to consume the entire available hugepage pool.
If they race instantiating the last page in the mapping, they will both
attempt to allocate the last available hugepage. One will fail, of course,
returning OOM from the fault and thus causing the process to be killed,
despite the fact that the entire mapping can, in fact, be instantiated.
The patch fixes this race by the simple method of adding a (sleeping) mutex
to serialize the hugepage fault path between allocation and insertion into
pagetables and/or page cache. It would be possible to avoid the
serialization by catching the allocation failures, waiting on some
condition, then rechecking to see if someone else has instantiated the page
for us. Given the likely frequency of hugepage instantiations, it seems
very doubtful it's worth the extra complexity.
This patch causes no regression on the libhugetlbfs testsuite, and one
test, which can trigger this race now passes where it previously failed.
Actually, the test still sometimes fails, though less often and only as a
shmat() failure, rather processes getting OOM killed by the VM. The dodgy
heuristic tests in fs/hugetlbfs/inode.c for whether there's enough hugepage
space aren't protected by the new mutex, and would be ugly to do so, so
there's still a race there. Another patch to replace those tests with
something saner for this reason as well as others coming...
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-03-22 15:08:53 +07:00
|
|
|
#include <linux/mutex.h>
|
2008-07-24 11:27:47 +07:00
|
|
|
#include <linux/bootmem.h>
|
2008-07-24 11:27:44 +07:00
|
|
|
#include <linux/sysfs.h>
|
2008-08-07 02:04:54 +07:00
|
|
|
|
2005-06-22 07:14:44 +07:00
|
|
|
#include <asm/page.h>
|
|
|
|
#include <asm/pgtable.h>
|
2008-07-29 05:46:30 +07:00
|
|
|
#include <asm/io.h>
|
2005-06-22 07:14:44 +07:00
|
|
|
|
|
|
|
#include <linux/hugetlb.h>
|
2006-03-22 15:08:40 +07:00
|
|
|
#include "internal.h"
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
const unsigned long hugetlb_zero = 0, hugetlb_infinity = ~0UL;
|
2007-07-17 18:03:13 +07:00
|
|
|
static gfp_t htlb_alloc_mask = GFP_HIGHUSER;
|
|
|
|
unsigned long hugepages_treat_as_movable;
|
2008-07-24 11:27:41 +07:00
|
|
|
|
2008-07-24 11:27:42 +07:00
|
|
|
static int max_hstate;
|
|
|
|
unsigned int default_hstate_idx;
|
|
|
|
struct hstate hstates[HUGE_MAX_HSTATE];
|
|
|
|
|
2008-07-24 11:27:52 +07:00
|
|
|
__initdata LIST_HEAD(huge_boot_pages);
|
|
|
|
|
2008-07-24 11:27:42 +07:00
|
|
|
/* for command line parsing */
|
|
|
|
static struct hstate * __initdata parsed_hstate;
|
|
|
|
static unsigned long __initdata default_hstate_max_huge_pages;
|
2008-07-24 11:27:52 +07:00
|
|
|
static unsigned long __initdata default_hstate_size;
|
2008-07-24 11:27:42 +07:00
|
|
|
|
|
|
|
#define for_each_hstate(h) \
|
|
|
|
for ((h) = hstates; (h) < &hstates[max_hstate]; (h)++)
|
2007-07-17 18:03:13 +07:00
|
|
|
|
[PATCH] hugepage: serialize hugepage allocation and instantiation
Currently, no lock or mutex is held between allocating a hugepage and
inserting it into the pagetables / page cache. When we do go to insert the
page into pagetables or page cache, we recheck and may free the newly
allocated hugepage. However, since the number of hugepages in the system
is strictly limited, and it's usualy to want to use all of them, this can
still lead to spurious allocation failures.
For example, suppose two processes are both mapping (MAP_SHARED) the same
hugepage file, large enough to consume the entire available hugepage pool.
If they race instantiating the last page in the mapping, they will both
attempt to allocate the last available hugepage. One will fail, of course,
returning OOM from the fault and thus causing the process to be killed,
despite the fact that the entire mapping can, in fact, be instantiated.
The patch fixes this race by the simple method of adding a (sleeping) mutex
to serialize the hugepage fault path between allocation and insertion into
pagetables and/or page cache. It would be possible to avoid the
serialization by catching the allocation failures, waiting on some
condition, then rechecking to see if someone else has instantiated the page
for us. Given the likely frequency of hugepage instantiations, it seems
very doubtful it's worth the extra complexity.
This patch causes no regression on the libhugetlbfs testsuite, and one
test, which can trigger this race now passes where it previously failed.
Actually, the test still sometimes fails, though less often and only as a
shmat() failure, rather processes getting OOM killed by the VM. The dodgy
heuristic tests in fs/hugetlbfs/inode.c for whether there's enough hugepage
space aren't protected by the new mutex, and would be ugly to do so, so
there's still a race there. Another patch to replace those tests with
something saner for this reason as well as others coming...
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-03-22 15:08:53 +07:00
|
|
|
/*
|
|
|
|
* Protects updates to hugepage_freelists, nr_huge_pages, and free_huge_pages
|
|
|
|
*/
|
|
|
|
static DEFINE_SPINLOCK(hugetlb_lock);
|
2005-11-22 12:32:28 +07:00
|
|
|
|
2008-07-24 11:27:29 +07:00
|
|
|
/*
|
|
|
|
* Region tracking -- allows tracking of reservations and instantiated pages
|
|
|
|
* across the pages in a mapping.
|
2008-07-24 11:27:32 +07:00
|
|
|
*
|
|
|
|
* The region data structures are protected by a combination of the mmap_sem
|
|
|
|
* and the hugetlb_instantion_mutex. To access or modify a region the caller
|
|
|
|
* must either hold the mmap_sem for write, or the mmap_sem for read and
|
|
|
|
* the hugetlb_instantiation mutex:
|
|
|
|
*
|
|
|
|
* down_write(&mm->mmap_sem);
|
|
|
|
* or
|
|
|
|
* down_read(&mm->mmap_sem);
|
|
|
|
* mutex_lock(&hugetlb_instantiation_mutex);
|
2008-07-24 11:27:29 +07:00
|
|
|
*/
|
|
|
|
struct file_region {
|
|
|
|
struct list_head link;
|
|
|
|
long from;
|
|
|
|
long to;
|
|
|
|
};
|
|
|
|
|
|
|
|
static long region_add(struct list_head *head, long f, long t)
|
|
|
|
{
|
|
|
|
struct file_region *rg, *nrg, *trg;
|
|
|
|
|
|
|
|
/* Locate the region we are either in or before. */
|
|
|
|
list_for_each_entry(rg, head, link)
|
|
|
|
if (f <= rg->to)
|
|
|
|
break;
|
|
|
|
|
|
|
|
/* Round our left edge to the current segment if it encloses us. */
|
|
|
|
if (f > rg->from)
|
|
|
|
f = rg->from;
|
|
|
|
|
|
|
|
/* Check for and consume any regions we now overlap with. */
|
|
|
|
nrg = rg;
|
|
|
|
list_for_each_entry_safe(rg, trg, rg->link.prev, link) {
|
|
|
|
if (&rg->link == head)
|
|
|
|
break;
|
|
|
|
if (rg->from > t)
|
|
|
|
break;
|
|
|
|
|
|
|
|
/* If this area reaches higher then extend our area to
|
|
|
|
* include it completely. If this is not the first area
|
|
|
|
* which we intend to reuse, free it. */
|
|
|
|
if (rg->to > t)
|
|
|
|
t = rg->to;
|
|
|
|
if (rg != nrg) {
|
|
|
|
list_del(&rg->link);
|
|
|
|
kfree(rg);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
nrg->from = f;
|
|
|
|
nrg->to = t;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static long region_chg(struct list_head *head, long f, long t)
|
|
|
|
{
|
|
|
|
struct file_region *rg, *nrg;
|
|
|
|
long chg = 0;
|
|
|
|
|
|
|
|
/* Locate the region we are before or in. */
|
|
|
|
list_for_each_entry(rg, head, link)
|
|
|
|
if (f <= rg->to)
|
|
|
|
break;
|
|
|
|
|
|
|
|
/* If we are below the current region then a new region is required.
|
|
|
|
* Subtle, allocate a new region at the position but make it zero
|
|
|
|
* size such that we can guarantee to record the reservation. */
|
|
|
|
if (&rg->link == head || t < rg->from) {
|
|
|
|
nrg = kmalloc(sizeof(*nrg), GFP_KERNEL);
|
|
|
|
if (!nrg)
|
|
|
|
return -ENOMEM;
|
|
|
|
nrg->from = f;
|
|
|
|
nrg->to = f;
|
|
|
|
INIT_LIST_HEAD(&nrg->link);
|
|
|
|
list_add(&nrg->link, rg->link.prev);
|
|
|
|
|
|
|
|
return t - f;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Round our left edge to the current segment if it encloses us. */
|
|
|
|
if (f > rg->from)
|
|
|
|
f = rg->from;
|
|
|
|
chg = t - f;
|
|
|
|
|
|
|
|
/* Check for and consume any regions we now overlap with. */
|
|
|
|
list_for_each_entry(rg, rg->link.prev, link) {
|
|
|
|
if (&rg->link == head)
|
|
|
|
break;
|
|
|
|
if (rg->from > t)
|
|
|
|
return chg;
|
|
|
|
|
|
|
|
/* We overlap with this area, if it extends futher than
|
|
|
|
* us then we must extend ourselves. Account for its
|
|
|
|
* existing reservation. */
|
|
|
|
if (rg->to > t) {
|
|
|
|
chg += rg->to - t;
|
|
|
|
t = rg->to;
|
|
|
|
}
|
|
|
|
chg -= rg->to - rg->from;
|
|
|
|
}
|
|
|
|
return chg;
|
|
|
|
}
|
|
|
|
|
|
|
|
static long region_truncate(struct list_head *head, long end)
|
|
|
|
{
|
|
|
|
struct file_region *rg, *trg;
|
|
|
|
long chg = 0;
|
|
|
|
|
|
|
|
/* Locate the region we are either in or before. */
|
|
|
|
list_for_each_entry(rg, head, link)
|
|
|
|
if (end <= rg->to)
|
|
|
|
break;
|
|
|
|
if (&rg->link == head)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/* If we are in the middle of a region then adjust it. */
|
|
|
|
if (end > rg->from) {
|
|
|
|
chg = rg->to - end;
|
|
|
|
rg->to = end;
|
|
|
|
rg = list_entry(rg->link.next, typeof(*rg), link);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Drop any remaining regions. */
|
|
|
|
list_for_each_entry_safe(rg, trg, rg->link.prev, link) {
|
|
|
|
if (&rg->link == head)
|
|
|
|
break;
|
|
|
|
chg += rg->to - rg->from;
|
|
|
|
list_del(&rg->link);
|
|
|
|
kfree(rg);
|
|
|
|
}
|
|
|
|
return chg;
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:32 +07:00
|
|
|
static long region_count(struct list_head *head, long f, long t)
|
|
|
|
{
|
|
|
|
struct file_region *rg;
|
|
|
|
long chg = 0;
|
|
|
|
|
|
|
|
/* Locate each segment we overlap with, and count that overlap. */
|
|
|
|
list_for_each_entry(rg, head, link) {
|
|
|
|
int seg_from;
|
|
|
|
int seg_to;
|
|
|
|
|
|
|
|
if (rg->to <= f)
|
|
|
|
continue;
|
|
|
|
if (rg->from >= t)
|
|
|
|
break;
|
|
|
|
|
|
|
|
seg_from = max(rg->from, f);
|
|
|
|
seg_to = min(rg->to, t);
|
|
|
|
|
|
|
|
chg += seg_to - seg_from;
|
|
|
|
}
|
|
|
|
|
|
|
|
return chg;
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:26 +07:00
|
|
|
/*
|
|
|
|
* Convert the address within this vma to the page offset within
|
|
|
|
* the mapping, in pagecache page units; huge pages here.
|
|
|
|
*/
|
2008-07-24 11:27:41 +07:00
|
|
|
static pgoff_t vma_hugecache_offset(struct hstate *h,
|
|
|
|
struct vm_area_struct *vma, unsigned long address)
|
2008-07-24 11:27:26 +07:00
|
|
|
{
|
2008-07-24 11:27:41 +07:00
|
|
|
return ((address - vma->vm_start) >> huge_page_shift(h)) +
|
|
|
|
(vma->vm_pgoff >> huge_page_order(h));
|
2008-07-24 11:27:26 +07:00
|
|
|
}
|
|
|
|
|
2009-01-07 05:38:53 +07:00
|
|
|
/*
|
|
|
|
* Return the size of the pages allocated when backing a VMA. In the majority
|
|
|
|
* cases this will be same size as used by the page table entries.
|
|
|
|
*/
|
|
|
|
unsigned long vma_kernel_pagesize(struct vm_area_struct *vma)
|
|
|
|
{
|
|
|
|
struct hstate *hstate;
|
|
|
|
|
|
|
|
if (!is_vm_hugetlb_page(vma))
|
|
|
|
return PAGE_SIZE;
|
|
|
|
|
|
|
|
hstate = hstate_vma(vma);
|
|
|
|
|
|
|
|
return 1UL << (hstate->order + PAGE_SHIFT);
|
|
|
|
}
|
|
|
|
|
2009-01-07 05:38:54 +07:00
|
|
|
/*
|
|
|
|
* Return the page size being used by the MMU to back a VMA. In the majority
|
|
|
|
* of cases, the page size used by the kernel matches the MMU size. On
|
|
|
|
* architectures where it differs, an architecture-specific version of this
|
|
|
|
* function is required.
|
|
|
|
*/
|
|
|
|
#ifndef vma_mmu_pagesize
|
|
|
|
unsigned long vma_mmu_pagesize(struct vm_area_struct *vma)
|
|
|
|
{
|
|
|
|
return vma_kernel_pagesize(vma);
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2008-07-24 11:27:32 +07:00
|
|
|
/*
|
|
|
|
* Flags for MAP_PRIVATE reservations. These are stored in the bottom
|
|
|
|
* bits of the reservation map pointer, which are always clear due to
|
|
|
|
* alignment.
|
|
|
|
*/
|
|
|
|
#define HPAGE_RESV_OWNER (1UL << 0)
|
|
|
|
#define HPAGE_RESV_UNMAPPED (1UL << 1)
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
#define HPAGE_RESV_MASK (HPAGE_RESV_OWNER | HPAGE_RESV_UNMAPPED)
|
2008-07-24 11:27:32 +07:00
|
|
|
|
2008-07-24 11:27:23 +07:00
|
|
|
/*
|
|
|
|
* These helpers are used to track how many pages are reserved for
|
|
|
|
* faults in a MAP_PRIVATE mapping. Only the process that called mmap()
|
|
|
|
* is guaranteed to have their future faults succeed.
|
|
|
|
*
|
|
|
|
* With the exception of reset_vma_resv_huge_pages() which is called at fork(),
|
|
|
|
* the reserve counters are updated with the hugetlb_lock held. It is safe
|
|
|
|
* to reset the VMA at fork() time as it is not in use yet and there is no
|
|
|
|
* chance of the global counters getting corrupted as a result of the values.
|
2008-07-24 11:27:32 +07:00
|
|
|
*
|
|
|
|
* The private mapping reservation is represented in a subtly different
|
|
|
|
* manner to a shared mapping. A shared mapping has a region map associated
|
|
|
|
* with the underlying file, this region map represents the backing file
|
|
|
|
* pages which have ever had a reservation assigned which this persists even
|
|
|
|
* after the page is instantiated. A private mapping has a region map
|
|
|
|
* associated with the original mmap which is attached to all VMAs which
|
|
|
|
* reference it, this region map represents those offsets which have consumed
|
|
|
|
* reservation ie. where pages have been instantiated.
|
2008-07-24 11:27:23 +07:00
|
|
|
*/
|
2008-07-24 11:27:26 +07:00
|
|
|
static unsigned long get_vma_private_data(struct vm_area_struct *vma)
|
|
|
|
{
|
|
|
|
return (unsigned long)vma->vm_private_data;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void set_vma_private_data(struct vm_area_struct *vma,
|
|
|
|
unsigned long value)
|
|
|
|
{
|
|
|
|
vma->vm_private_data = (void *)value;
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:32 +07:00
|
|
|
struct resv_map {
|
|
|
|
struct kref refs;
|
|
|
|
struct list_head regions;
|
|
|
|
};
|
|
|
|
|
2008-10-19 10:27:06 +07:00
|
|
|
static struct resv_map *resv_map_alloc(void)
|
2008-07-24 11:27:32 +07:00
|
|
|
{
|
|
|
|
struct resv_map *resv_map = kmalloc(sizeof(*resv_map), GFP_KERNEL);
|
|
|
|
if (!resv_map)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
kref_init(&resv_map->refs);
|
|
|
|
INIT_LIST_HEAD(&resv_map->regions);
|
|
|
|
|
|
|
|
return resv_map;
|
|
|
|
}
|
|
|
|
|
2008-10-19 10:27:06 +07:00
|
|
|
static void resv_map_release(struct kref *ref)
|
2008-07-24 11:27:32 +07:00
|
|
|
{
|
|
|
|
struct resv_map *resv_map = container_of(ref, struct resv_map, refs);
|
|
|
|
|
|
|
|
/* Clear out any active regions before we release the map. */
|
|
|
|
region_truncate(&resv_map->regions, 0);
|
|
|
|
kfree(resv_map);
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct resv_map *vma_resv_map(struct vm_area_struct *vma)
|
2008-07-24 11:27:23 +07:00
|
|
|
{
|
|
|
|
VM_BUG_ON(!is_vm_hugetlb_page(vma));
|
2009-05-29 04:34:40 +07:00
|
|
|
if (!(vma->vm_flags & VM_MAYSHARE))
|
2008-07-24 11:27:32 +07:00
|
|
|
return (struct resv_map *)(get_vma_private_data(vma) &
|
|
|
|
~HPAGE_RESV_MASK);
|
2008-10-19 10:27:06 +07:00
|
|
|
return NULL;
|
2008-07-24 11:27:23 +07:00
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:32 +07:00
|
|
|
static void set_vma_resv_map(struct vm_area_struct *vma, struct resv_map *map)
|
2008-07-24 11:27:23 +07:00
|
|
|
{
|
|
|
|
VM_BUG_ON(!is_vm_hugetlb_page(vma));
|
2009-05-29 04:34:40 +07:00
|
|
|
VM_BUG_ON(vma->vm_flags & VM_MAYSHARE);
|
2008-07-24 11:27:23 +07:00
|
|
|
|
2008-07-24 11:27:32 +07:00
|
|
|
set_vma_private_data(vma, (get_vma_private_data(vma) &
|
|
|
|
HPAGE_RESV_MASK) | (unsigned long)map);
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
static void set_vma_resv_flags(struct vm_area_struct *vma, unsigned long flags)
|
|
|
|
{
|
|
|
|
VM_BUG_ON(!is_vm_hugetlb_page(vma));
|
2009-05-29 04:34:40 +07:00
|
|
|
VM_BUG_ON(vma->vm_flags & VM_MAYSHARE);
|
2008-07-24 11:27:26 +07:00
|
|
|
|
|
|
|
set_vma_private_data(vma, get_vma_private_data(vma) | flags);
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
static int is_vma_resv_set(struct vm_area_struct *vma, unsigned long flag)
|
|
|
|
{
|
|
|
|
VM_BUG_ON(!is_vm_hugetlb_page(vma));
|
2008-07-24 11:27:26 +07:00
|
|
|
|
|
|
|
return (get_vma_private_data(vma) & flag) != 0;
|
2008-07-24 11:27:23 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Decrement the reserved pages in the hugepage pool by one */
|
2008-07-24 11:27:41 +07:00
|
|
|
static void decrement_hugepage_resv_vma(struct hstate *h,
|
|
|
|
struct vm_area_struct *vma)
|
2008-07-24 11:27:23 +07:00
|
|
|
{
|
2008-07-24 11:27:30 +07:00
|
|
|
if (vma->vm_flags & VM_NORESERVE)
|
|
|
|
return;
|
|
|
|
|
2009-05-29 04:34:40 +07:00
|
|
|
if (vma->vm_flags & VM_MAYSHARE) {
|
2008-07-24 11:27:23 +07:00
|
|
|
/* Shared mappings always use reserves */
|
2008-07-24 11:27:41 +07:00
|
|
|
h->resv_huge_pages--;
|
2008-07-24 11:27:32 +07:00
|
|
|
} else if (is_vma_resv_set(vma, HPAGE_RESV_OWNER)) {
|
2008-07-24 11:27:23 +07:00
|
|
|
/*
|
|
|
|
* Only the process that called mmap() has reserves for
|
|
|
|
* private mappings.
|
|
|
|
*/
|
2008-07-24 11:27:41 +07:00
|
|
|
h->resv_huge_pages--;
|
2008-07-24 11:27:23 +07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
/* Reset counters to 0 and clear all HPAGE_RESV_* flags */
|
2008-07-24 11:27:23 +07:00
|
|
|
void reset_vma_resv_huge_pages(struct vm_area_struct *vma)
|
|
|
|
{
|
|
|
|
VM_BUG_ON(!is_vm_hugetlb_page(vma));
|
2009-05-29 04:34:40 +07:00
|
|
|
if (!(vma->vm_flags & VM_MAYSHARE))
|
2008-07-24 11:27:23 +07:00
|
|
|
vma->vm_private_data = (void *)0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Returns true if the VMA has associated reserve pages */
|
2008-07-24 11:27:58 +07:00
|
|
|
static int vma_has_reserves(struct vm_area_struct *vma)
|
2008-07-24 11:27:23 +07:00
|
|
|
{
|
2009-05-29 04:34:40 +07:00
|
|
|
if (vma->vm_flags & VM_MAYSHARE)
|
2008-07-24 11:27:58 +07:00
|
|
|
return 1;
|
|
|
|
if (is_vma_resv_set(vma, HPAGE_RESV_OWNER))
|
|
|
|
return 1;
|
|
|
|
return 0;
|
2008-07-24 11:27:23 +07:00
|
|
|
}
|
|
|
|
|
2008-11-07 03:53:26 +07:00
|
|
|
static void clear_gigantic_page(struct page *page,
|
|
|
|
unsigned long addr, unsigned long sz)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
struct page *p = page;
|
|
|
|
|
|
|
|
might_sleep();
|
|
|
|
for (i = 0; i < sz/PAGE_SIZE; i++, p = mem_map_next(p, page, i)) {
|
|
|
|
cond_resched();
|
|
|
|
clear_user_highpage(p, addr + i * PAGE_SIZE);
|
|
|
|
}
|
|
|
|
}
|
2008-07-24 11:27:41 +07:00
|
|
|
static void clear_huge_page(struct page *page,
|
|
|
|
unsigned long addr, unsigned long sz)
|
2006-03-22 15:08:51 +07:00
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
2009-01-07 05:39:58 +07:00
|
|
|
if (unlikely(sz > MAX_ORDER_NR_PAGES)) {
|
|
|
|
clear_gigantic_page(page, addr, sz);
|
|
|
|
return;
|
|
|
|
}
|
2008-11-07 03:53:26 +07:00
|
|
|
|
2006-03-22 15:08:51 +07:00
|
|
|
might_sleep();
|
2008-07-24 11:27:41 +07:00
|
|
|
for (i = 0; i < sz/PAGE_SIZE; i++) {
|
2006-03-22 15:08:51 +07:00
|
|
|
cond_resched();
|
2007-10-01 15:20:10 +07:00
|
|
|
clear_user_highpage(page + i, addr + i * PAGE_SIZE);
|
2006-03-22 15:08:51 +07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-11-07 03:53:26 +07:00
|
|
|
static void copy_gigantic_page(struct page *dst, struct page *src,
|
|
|
|
unsigned long addr, struct vm_area_struct *vma)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
struct hstate *h = hstate_vma(vma);
|
|
|
|
struct page *dst_base = dst;
|
|
|
|
struct page *src_base = src;
|
|
|
|
might_sleep();
|
|
|
|
for (i = 0; i < pages_per_huge_page(h); ) {
|
|
|
|
cond_resched();
|
|
|
|
copy_user_highpage(dst, src, addr + i*PAGE_SIZE, vma);
|
|
|
|
|
|
|
|
i++;
|
|
|
|
dst = mem_map_next(dst, dst_base, i);
|
|
|
|
src = mem_map_next(src, src_base, i);
|
|
|
|
}
|
|
|
|
}
|
2006-03-22 15:08:51 +07:00
|
|
|
static void copy_huge_page(struct page *dst, struct page *src,
|
2006-12-13 00:14:55 +07:00
|
|
|
unsigned long addr, struct vm_area_struct *vma)
|
2006-03-22 15:08:51 +07:00
|
|
|
{
|
|
|
|
int i;
|
2008-07-24 11:27:41 +07:00
|
|
|
struct hstate *h = hstate_vma(vma);
|
2006-03-22 15:08:51 +07:00
|
|
|
|
2009-01-07 05:39:58 +07:00
|
|
|
if (unlikely(pages_per_huge_page(h) > MAX_ORDER_NR_PAGES)) {
|
|
|
|
copy_gigantic_page(dst, src, addr, vma);
|
|
|
|
return;
|
|
|
|
}
|
2008-11-07 03:53:26 +07:00
|
|
|
|
2006-03-22 15:08:51 +07:00
|
|
|
might_sleep();
|
2008-07-24 11:27:41 +07:00
|
|
|
for (i = 0; i < pages_per_huge_page(h); i++) {
|
2006-03-22 15:08:51 +07:00
|
|
|
cond_resched();
|
2006-12-13 00:14:55 +07:00
|
|
|
copy_user_highpage(dst + i, src + i, addr + i*PAGE_SIZE, vma);
|
2006-03-22 15:08:51 +07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
static void enqueue_huge_page(struct hstate *h, struct page *page)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
int nid = page_to_nid(page);
|
2008-07-24 11:27:41 +07:00
|
|
|
list_add(&page->lru, &h->hugepage_freelists[nid]);
|
|
|
|
h->free_huge_pages++;
|
|
|
|
h->free_huge_pages_node[nid]++;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
static struct page *dequeue_huge_page(struct hstate *h)
|
2008-03-05 05:29:42 +07:00
|
|
|
{
|
|
|
|
int nid;
|
|
|
|
struct page *page = NULL;
|
|
|
|
|
|
|
|
for (nid = 0; nid < MAX_NUMNODES; ++nid) {
|
2008-07-24 11:27:41 +07:00
|
|
|
if (!list_empty(&h->hugepage_freelists[nid])) {
|
|
|
|
page = list_entry(h->hugepage_freelists[nid].next,
|
2008-03-05 05:29:42 +07:00
|
|
|
struct page, lru);
|
|
|
|
list_del(&page->lru);
|
2008-07-24 11:27:41 +07:00
|
|
|
h->free_huge_pages--;
|
|
|
|
h->free_huge_pages_node[nid]--;
|
2008-03-05 05:29:42 +07:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return page;
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
static struct page *dequeue_huge_page_vma(struct hstate *h,
|
|
|
|
struct vm_area_struct *vma,
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
unsigned long address, int avoid_reserve)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2007-07-16 13:38:02 +07:00
|
|
|
int nid;
|
2005-04-17 05:20:36 +07:00
|
|
|
struct page *page = NULL;
|
Fix NUMA Memory Policy Reference Counting
This patch proposes fixes to the reference counting of memory policy in the
page allocation paths and in show_numa_map(). Extracted from my "Memory
Policy Cleanups and Enhancements" series as stand-alone.
Shared policy lookup [shmem] has always added a reference to the policy,
but this was never unrefed after page allocation or after formatting the
numa map data.
Default system policy should not require additional ref counting, nor
should the current task's task policy. However, show_numa_map() calls
get_vma_policy() to examine what may be [likely is] another task's policy.
The latter case needs protection against freeing of the policy.
This patch adds a reference count to a mempolicy returned by
get_vma_policy() when the policy is a vma policy or another task's
mempolicy. Again, shared policy is already reference counted on lookup. A
matching "unref" [__mpol_free()] is performed in alloc_page_vma() for
shared and vma policies, and in show_numa_map() for shared and another
task's mempolicy. We can call __mpol_free() directly, saving an admittedly
inexpensive inline NULL test, because we know we have a non-NULL policy.
Handling policy ref counts for hugepages is a bit trickier.
huge_zonelist() returns a zone list that might come from a shared or vma
'BIND policy. In this case, we should hold the reference until after the
huge page allocation in dequeue_hugepage(). The patch modifies
huge_zonelist() to return a pointer to the mempolicy if it needs to be
unref'd after allocation.
Kernel Build [16cpu, 32GB, ia64] - average of 10 runs:
w/o patch w/ refcount patch
Avg Std Devn Avg Std Devn
Real: 100.59 0.38 100.63 0.43
User: 1209.60 0.37 1209.91 0.31
System: 81.52 0.42 81.64 0.34
Signed-off-by: Lee Schermerhorn <lee.schermerhorn@hp.com>
Acked-by: Andi Kleen <ak@suse.de>
Cc: Christoph Lameter <clameter@sgi.com>
Acked-by: Mel Gorman <mel@csn.ul.ie>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-09-19 12:46:47 +07:00
|
|
|
struct mempolicy *mpol;
|
2008-04-28 16:12:18 +07:00
|
|
|
nodemask_t *nodemask;
|
2007-07-17 18:03:13 +07:00
|
|
|
struct zonelist *zonelist = huge_zonelist(vma, address,
|
2008-04-28 16:12:18 +07:00
|
|
|
htlb_alloc_mask, &mpol, &nodemask);
|
2008-04-28 16:12:17 +07:00
|
|
|
struct zone *zone;
|
|
|
|
struct zoneref *z;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2008-07-24 11:27:23 +07:00
|
|
|
/*
|
|
|
|
* A child process with MAP_PRIVATE mappings created by their parent
|
|
|
|
* have no page reserves. This check ensures that reservations are
|
|
|
|
* not "stolen". The child may still get SIGKILLed
|
|
|
|
*/
|
2008-07-24 11:27:58 +07:00
|
|
|
if (!vma_has_reserves(vma) &&
|
2008-07-24 11:27:41 +07:00
|
|
|
h->free_huge_pages - h->resv_huge_pages == 0)
|
2008-07-24 11:27:23 +07:00
|
|
|
return NULL;
|
|
|
|
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
/* If reserves cannot be used, ensure enough pages are in the pool */
|
2008-07-24 11:27:41 +07:00
|
|
|
if (avoid_reserve && h->free_huge_pages - h->resv_huge_pages == 0)
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
return NULL;
|
|
|
|
|
2008-04-28 16:12:18 +07:00
|
|
|
for_each_zone_zonelist_nodemask(zone, z, zonelist,
|
|
|
|
MAX_NR_ZONES - 1, nodemask) {
|
2008-04-28 16:12:16 +07:00
|
|
|
nid = zone_to_nid(zone);
|
|
|
|
if (cpuset_zone_allowed_softwall(zone, htlb_alloc_mask) &&
|
2008-07-24 11:27:41 +07:00
|
|
|
!list_empty(&h->hugepage_freelists[nid])) {
|
|
|
|
page = list_entry(h->hugepage_freelists[nid].next,
|
2007-07-19 15:49:08 +07:00
|
|
|
struct page, lru);
|
|
|
|
list_del(&page->lru);
|
2008-07-24 11:27:41 +07:00
|
|
|
h->free_huge_pages--;
|
|
|
|
h->free_huge_pages_node[nid]--;
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
|
|
|
|
if (!avoid_reserve)
|
2008-07-24 11:27:41 +07:00
|
|
|
decrement_hugepage_resv_vma(h, vma);
|
2008-07-24 11:27:23 +07:00
|
|
|
|
2007-07-24 08:44:00 +07:00
|
|
|
break;
|
2007-07-19 15:49:08 +07:00
|
|
|
}
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
mempolicy: rework mempolicy Reference Counting [yet again]
After further discussion with Christoph Lameter, it has become clear that my
earlier attempts to clean up the mempolicy reference counting were a bit of
overkill in some areas, resulting in superflous ref/unref in what are usually
fast paths. In other areas, further inspection reveals that I botched the
unref for interleave policies.
A separate patch, suitable for upstream/stable trees, fixes up the known
errors in the previous attempt to fix reference counting.
This patch reworks the memory policy referencing counting and, one hopes,
simplifies the code. Maybe I'll get it right this time.
See the update to the numa_memory_policy.txt document for a discussion of
memory policy reference counting that motivates this patch.
Summary:
Lookup of mempolicy, based on (vma, address) need only add a reference for
shared policy, and we need only unref the policy when finished for shared
policies. So, this patch backs out all of the unneeded extra reference
counting added by my previous attempt. It then unrefs only shared policies
when we're finished with them, using the mpol_cond_put() [conditional put]
helper function introduced by this patch.
Note that shmem_swapin() calls read_swap_cache_async() with a dummy vma
containing just the policy. read_swap_cache_async() can call alloc_page_vma()
multiple times, so we can't let alloc_page_vma() unref the shared policy in
this case. To avoid this, we make a copy of any non-null shared policy and
remove the MPOL_F_SHARED flag from the copy. This copy occurs before reading
a page [or multiple pages] from swap, so the overhead should not be an issue
here.
I introduced a new static inline function "mpol_cond_copy()" to copy the
shared policy to an on-stack policy and remove the flags that would require a
conditional free. The current implementation of mpol_cond_copy() assumes that
the struct mempolicy contains no pointers to dynamically allocated structures
that must be duplicated or reference counted during copy.
Signed-off-by: Lee Schermerhorn <lee.schermerhorn@hp.com>
Cc: Christoph Lameter <clameter@sgi.com>
Cc: David Rientjes <rientjes@google.com>
Cc: Mel Gorman <mel@csn.ul.ie>
Cc: Andi Kleen <ak@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-04-28 16:13:16 +07:00
|
|
|
mpol_cond_put(mpol);
|
2005-04-17 05:20:36 +07:00
|
|
|
return page;
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
static void update_and_free_page(struct hstate *h, struct page *page)
|
hugetlb: Move update_and_free_page
Dynamic huge page pool resizing.
In most real-world scenarios, configuring the size of the hugetlb pool
correctly is a difficult task. If too few pages are allocated to the pool,
applications using MAP_SHARED may fail to mmap() a hugepage region and
applications using MAP_PRIVATE may receive SIGBUS. Isolating too much memory
in the hugetlb pool means it is not available for other uses, especially those
programs not using huge pages.
The obvious answer is to let the hugetlb pool grow and shrink in response to
the runtime demand for huge pages. The work Mel Gorman has been doing to
establish a memory zone for movable memory allocations makes dynamically
resizing the hugetlb pool reliable within the limits of that zone. This patch
series implements dynamic pool resizing for private and shared mappings while
being careful to maintain existing semantics. Please reply with your comments
and feedback; even just to say whether it would be a useful feature to you.
Thanks.
How it works
============
Upon depletion of the hugetlb pool, rather than reporting an error immediately,
first try and allocate the needed huge pages directly from the buddy allocator.
Care must be taken to avoid unbounded growth of the hugetlb pool, so the
hugetlb filesystem quota is used to limit overall pool size.
The real work begins when we decide there is a shortage of huge pages. What
happens next depends on whether the pages are for a private or shared mapping.
Private mappings are straightforward. At fault time, if alloc_huge_page()
fails, we allocate a page from the buddy allocator and increment the source
node's surplus_huge_pages counter. When free_huge_page() is called for a page
on a node with a surplus, the page is freed directly to the buddy allocator
instead of the hugetlb pool.
Because shared mappings require all of the pages to be reserved up front, some
additional work must be done at mmap() to support them. We determine the
reservation shortage and allocate the required number of pages all at once.
These pages are then added to the hugetlb pool and marked reserved. Where that
is not possible the mmap() will fail. As with private mappings, the
appropriate surplus counters are updated. Since reserved huge pages won't
necessarily be used by the process, we can't be sure that free_huge_page() will
always be called to return surplus pages to the buddy allocator. To prevent
the huge page pool from bloating, we must free unused surplus pages when their
reservation has ended.
Controlling it
==============
With the entire patch series applied, pool resizing is off by default so unless
specific action is taken, the semantics are unchanged.
To take advantage of the flexibility afforded by this patch series one must
tolerate a change in semantics. To control hugetlb pool growth, the following
techniques can be employed:
* A sysctl tunable to enable/disable the feature entirely
* The size= mount option for hugetlbfs filesystems to limit pool size
Performance
===========
When contiguous memory is readily available, it is expected that the cost of
dynamicly resizing the pool will be small. This series has been performance
tested with 'stream' to measure this cost.
Stream (http://www.cs.virginia.edu/stream/) was linked with libhugetlbfs to
enable remapping of the text and data/bss segments into huge pages.
Stream with small array
-----------------------
Baseline: nr_hugepages = 0, No libhugetlbfs segment remapping
Preallocated: nr_hugepages = 5, Text and data/bss remapping
Dynamic: nr_hugepages = 0, Text and data/bss remapping
Rate (MB/s)
Function Baseline Preallocated Dynamic
Copy: 4695.6266 5942.8371 5982.2287
Scale: 4451.5776 5017.1419 5658.7843
Add: 5815.8849 7927.7827 8119.3552
Triad: 5949.4144 8527.6492 8110.6903
Stream with large array
-----------------------
Baseline: nr_hugepages = 0, No libhugetlbfs segment remapping
Preallocated: nr_hugepages = 67, Text and data/bss remapping
Dynamic: nr_hugepages = 0, Text and data/bss remapping
Rate (MB/s)
Function Baseline Preallocated Dynamic
Copy: 2227.8281 2544.2732 2546.4947
Scale: 2136.3208 2430.7294 2421.2074
Add: 2773.1449 4004.0021 3999.4331
Triad: 2748.4502 3777.0109 3773.4970
* All numbers are averages taken from 10 consecutive runs with a maximum
standard deviation of 1.3 percent noted.
This patch:
Simply move update_and_free_page() so that it can be reused later in this
patch series. The implementation is not changed.
Signed-off-by: Adam Litke <agl@us.ibm.com>
Acked-by: Andy Whitcroft <apw@shadowen.org>
Acked-by: Dave McCracken <dave.mccracken@oracle.com>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: David Gibson <david@gibson.dropbear.id.au>
Cc: Ken Chen <kenchen@google.com>
Cc: Badari Pulavarty <pbadari@us.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-16 15:26:16 +07:00
|
|
|
{
|
|
|
|
int i;
|
2008-07-24 11:27:41 +07:00
|
|
|
|
2008-11-07 03:53:27 +07:00
|
|
|
VM_BUG_ON(h->order >= MAX_ORDER);
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
h->nr_huge_pages--;
|
|
|
|
h->nr_huge_pages_node[page_to_nid(page)]--;
|
|
|
|
for (i = 0; i < pages_per_huge_page(h); i++) {
|
hugetlb: Move update_and_free_page
Dynamic huge page pool resizing.
In most real-world scenarios, configuring the size of the hugetlb pool
correctly is a difficult task. If too few pages are allocated to the pool,
applications using MAP_SHARED may fail to mmap() a hugepage region and
applications using MAP_PRIVATE may receive SIGBUS. Isolating too much memory
in the hugetlb pool means it is not available for other uses, especially those
programs not using huge pages.
The obvious answer is to let the hugetlb pool grow and shrink in response to
the runtime demand for huge pages. The work Mel Gorman has been doing to
establish a memory zone for movable memory allocations makes dynamically
resizing the hugetlb pool reliable within the limits of that zone. This patch
series implements dynamic pool resizing for private and shared mappings while
being careful to maintain existing semantics. Please reply with your comments
and feedback; even just to say whether it would be a useful feature to you.
Thanks.
How it works
============
Upon depletion of the hugetlb pool, rather than reporting an error immediately,
first try and allocate the needed huge pages directly from the buddy allocator.
Care must be taken to avoid unbounded growth of the hugetlb pool, so the
hugetlb filesystem quota is used to limit overall pool size.
The real work begins when we decide there is a shortage of huge pages. What
happens next depends on whether the pages are for a private or shared mapping.
Private mappings are straightforward. At fault time, if alloc_huge_page()
fails, we allocate a page from the buddy allocator and increment the source
node's surplus_huge_pages counter. When free_huge_page() is called for a page
on a node with a surplus, the page is freed directly to the buddy allocator
instead of the hugetlb pool.
Because shared mappings require all of the pages to be reserved up front, some
additional work must be done at mmap() to support them. We determine the
reservation shortage and allocate the required number of pages all at once.
These pages are then added to the hugetlb pool and marked reserved. Where that
is not possible the mmap() will fail. As with private mappings, the
appropriate surplus counters are updated. Since reserved huge pages won't
necessarily be used by the process, we can't be sure that free_huge_page() will
always be called to return surplus pages to the buddy allocator. To prevent
the huge page pool from bloating, we must free unused surplus pages when their
reservation has ended.
Controlling it
==============
With the entire patch series applied, pool resizing is off by default so unless
specific action is taken, the semantics are unchanged.
To take advantage of the flexibility afforded by this patch series one must
tolerate a change in semantics. To control hugetlb pool growth, the following
techniques can be employed:
* A sysctl tunable to enable/disable the feature entirely
* The size= mount option for hugetlbfs filesystems to limit pool size
Performance
===========
When contiguous memory is readily available, it is expected that the cost of
dynamicly resizing the pool will be small. This series has been performance
tested with 'stream' to measure this cost.
Stream (http://www.cs.virginia.edu/stream/) was linked with libhugetlbfs to
enable remapping of the text and data/bss segments into huge pages.
Stream with small array
-----------------------
Baseline: nr_hugepages = 0, No libhugetlbfs segment remapping
Preallocated: nr_hugepages = 5, Text and data/bss remapping
Dynamic: nr_hugepages = 0, Text and data/bss remapping
Rate (MB/s)
Function Baseline Preallocated Dynamic
Copy: 4695.6266 5942.8371 5982.2287
Scale: 4451.5776 5017.1419 5658.7843
Add: 5815.8849 7927.7827 8119.3552
Triad: 5949.4144 8527.6492 8110.6903
Stream with large array
-----------------------
Baseline: nr_hugepages = 0, No libhugetlbfs segment remapping
Preallocated: nr_hugepages = 67, Text and data/bss remapping
Dynamic: nr_hugepages = 0, Text and data/bss remapping
Rate (MB/s)
Function Baseline Preallocated Dynamic
Copy: 2227.8281 2544.2732 2546.4947
Scale: 2136.3208 2430.7294 2421.2074
Add: 2773.1449 4004.0021 3999.4331
Triad: 2748.4502 3777.0109 3773.4970
* All numbers are averages taken from 10 consecutive runs with a maximum
standard deviation of 1.3 percent noted.
This patch:
Simply move update_and_free_page() so that it can be reused later in this
patch series. The implementation is not changed.
Signed-off-by: Adam Litke <agl@us.ibm.com>
Acked-by: Andy Whitcroft <apw@shadowen.org>
Acked-by: Dave McCracken <dave.mccracken@oracle.com>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: David Gibson <david@gibson.dropbear.id.au>
Cc: Ken Chen <kenchen@google.com>
Cc: Badari Pulavarty <pbadari@us.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-16 15:26:16 +07:00
|
|
|
page[i].flags &= ~(1 << PG_locked | 1 << PG_error | 1 << PG_referenced |
|
|
|
|
1 << PG_dirty | 1 << PG_active | 1 << PG_reserved |
|
|
|
|
1 << PG_private | 1<< PG_writeback);
|
|
|
|
}
|
|
|
|
set_compound_page_dtor(page, NULL);
|
|
|
|
set_page_refcounted(page);
|
2008-04-28 16:13:29 +07:00
|
|
|
arch_release_hugepage(page);
|
2008-07-24 11:27:41 +07:00
|
|
|
__free_pages(page, huge_page_order(h));
|
hugetlb: Move update_and_free_page
Dynamic huge page pool resizing.
In most real-world scenarios, configuring the size of the hugetlb pool
correctly is a difficult task. If too few pages are allocated to the pool,
applications using MAP_SHARED may fail to mmap() a hugepage region and
applications using MAP_PRIVATE may receive SIGBUS. Isolating too much memory
in the hugetlb pool means it is not available for other uses, especially those
programs not using huge pages.
The obvious answer is to let the hugetlb pool grow and shrink in response to
the runtime demand for huge pages. The work Mel Gorman has been doing to
establish a memory zone for movable memory allocations makes dynamically
resizing the hugetlb pool reliable within the limits of that zone. This patch
series implements dynamic pool resizing for private and shared mappings while
being careful to maintain existing semantics. Please reply with your comments
and feedback; even just to say whether it would be a useful feature to you.
Thanks.
How it works
============
Upon depletion of the hugetlb pool, rather than reporting an error immediately,
first try and allocate the needed huge pages directly from the buddy allocator.
Care must be taken to avoid unbounded growth of the hugetlb pool, so the
hugetlb filesystem quota is used to limit overall pool size.
The real work begins when we decide there is a shortage of huge pages. What
happens next depends on whether the pages are for a private or shared mapping.
Private mappings are straightforward. At fault time, if alloc_huge_page()
fails, we allocate a page from the buddy allocator and increment the source
node's surplus_huge_pages counter. When free_huge_page() is called for a page
on a node with a surplus, the page is freed directly to the buddy allocator
instead of the hugetlb pool.
Because shared mappings require all of the pages to be reserved up front, some
additional work must be done at mmap() to support them. We determine the
reservation shortage and allocate the required number of pages all at once.
These pages are then added to the hugetlb pool and marked reserved. Where that
is not possible the mmap() will fail. As with private mappings, the
appropriate surplus counters are updated. Since reserved huge pages won't
necessarily be used by the process, we can't be sure that free_huge_page() will
always be called to return surplus pages to the buddy allocator. To prevent
the huge page pool from bloating, we must free unused surplus pages when their
reservation has ended.
Controlling it
==============
With the entire patch series applied, pool resizing is off by default so unless
specific action is taken, the semantics are unchanged.
To take advantage of the flexibility afforded by this patch series one must
tolerate a change in semantics. To control hugetlb pool growth, the following
techniques can be employed:
* A sysctl tunable to enable/disable the feature entirely
* The size= mount option for hugetlbfs filesystems to limit pool size
Performance
===========
When contiguous memory is readily available, it is expected that the cost of
dynamicly resizing the pool will be small. This series has been performance
tested with 'stream' to measure this cost.
Stream (http://www.cs.virginia.edu/stream/) was linked with libhugetlbfs to
enable remapping of the text and data/bss segments into huge pages.
Stream with small array
-----------------------
Baseline: nr_hugepages = 0, No libhugetlbfs segment remapping
Preallocated: nr_hugepages = 5, Text and data/bss remapping
Dynamic: nr_hugepages = 0, Text and data/bss remapping
Rate (MB/s)
Function Baseline Preallocated Dynamic
Copy: 4695.6266 5942.8371 5982.2287
Scale: 4451.5776 5017.1419 5658.7843
Add: 5815.8849 7927.7827 8119.3552
Triad: 5949.4144 8527.6492 8110.6903
Stream with large array
-----------------------
Baseline: nr_hugepages = 0, No libhugetlbfs segment remapping
Preallocated: nr_hugepages = 67, Text and data/bss remapping
Dynamic: nr_hugepages = 0, Text and data/bss remapping
Rate (MB/s)
Function Baseline Preallocated Dynamic
Copy: 2227.8281 2544.2732 2546.4947
Scale: 2136.3208 2430.7294 2421.2074
Add: 2773.1449 4004.0021 3999.4331
Triad: 2748.4502 3777.0109 3773.4970
* All numbers are averages taken from 10 consecutive runs with a maximum
standard deviation of 1.3 percent noted.
This patch:
Simply move update_and_free_page() so that it can be reused later in this
patch series. The implementation is not changed.
Signed-off-by: Adam Litke <agl@us.ibm.com>
Acked-by: Andy Whitcroft <apw@shadowen.org>
Acked-by: Dave McCracken <dave.mccracken@oracle.com>
Acked-by: William Irwin <bill.irwin@oracle.com>
Cc: David Gibson <david@gibson.dropbear.id.au>
Cc: Ken Chen <kenchen@google.com>
Cc: Badari Pulavarty <pbadari@us.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-16 15:26:16 +07:00
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:42 +07:00
|
|
|
struct hstate *size_to_hstate(unsigned long size)
|
|
|
|
{
|
|
|
|
struct hstate *h;
|
|
|
|
|
|
|
|
for_each_hstate(h) {
|
|
|
|
if (huge_page_size(h) == size)
|
|
|
|
return h;
|
|
|
|
}
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2006-03-22 15:08:56 +07:00
|
|
|
static void free_huge_page(struct page *page)
|
|
|
|
{
|
2008-07-24 11:27:41 +07:00
|
|
|
/*
|
|
|
|
* Can't pass hstate in here because it is called from the
|
|
|
|
* compound page destructor.
|
|
|
|
*/
|
2008-07-24 11:27:42 +07:00
|
|
|
struct hstate *h = page_hstate(page);
|
2007-10-16 15:26:18 +07:00
|
|
|
int nid = page_to_nid(page);
|
2007-11-15 07:59:38 +07:00
|
|
|
struct address_space *mapping;
|
2006-03-22 15:08:56 +07:00
|
|
|
|
2007-11-15 07:59:38 +07:00
|
|
|
mapping = (struct address_space *) page_private(page);
|
2008-02-24 06:23:32 +07:00
|
|
|
set_page_private(page, 0);
|
2007-10-16 15:26:18 +07:00
|
|
|
BUG_ON(page_count(page));
|
2006-03-22 15:08:56 +07:00
|
|
|
INIT_LIST_HEAD(&page->lru);
|
|
|
|
|
|
|
|
spin_lock(&hugetlb_lock);
|
2008-07-24 11:27:47 +07:00
|
|
|
if (h->surplus_huge_pages_node[nid] && huge_page_order(h) < MAX_ORDER) {
|
2008-07-24 11:27:41 +07:00
|
|
|
update_and_free_page(h, page);
|
|
|
|
h->surplus_huge_pages--;
|
|
|
|
h->surplus_huge_pages_node[nid]--;
|
2007-10-16 15:26:18 +07:00
|
|
|
} else {
|
2008-07-24 11:27:41 +07:00
|
|
|
enqueue_huge_page(h, page);
|
2007-10-16 15:26:18 +07:00
|
|
|
}
|
2006-03-22 15:08:56 +07:00
|
|
|
spin_unlock(&hugetlb_lock);
|
2007-11-15 07:59:38 +07:00
|
|
|
if (mapping)
|
2007-11-15 07:59:41 +07:00
|
|
|
hugetlb_put_quota(mapping, 1);
|
2006-03-22 15:08:56 +07:00
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
static void prep_new_huge_page(struct hstate *h, struct page *page, int nid)
|
2008-07-24 11:27:40 +07:00
|
|
|
{
|
|
|
|
set_compound_page_dtor(page, free_huge_page);
|
|
|
|
spin_lock(&hugetlb_lock);
|
2008-07-24 11:27:41 +07:00
|
|
|
h->nr_huge_pages++;
|
|
|
|
h->nr_huge_pages_node[nid]++;
|
2008-07-24 11:27:40 +07:00
|
|
|
spin_unlock(&hugetlb_lock);
|
|
|
|
put_page(page); /* free it into the hugepage allocator */
|
|
|
|
}
|
|
|
|
|
mm: introduce PageHuge() for testing huge/gigantic pages
A series of patches to enhance the /proc/pagemap interface and to add a
userspace executable which can be used to present the pagemap data.
Export 10 more flags to end users (and more for kernel developers):
11. KPF_MMAP (pseudo flag) memory mapped page
12. KPF_ANON (pseudo flag) memory mapped page (anonymous)
13. KPF_SWAPCACHE page is in swap cache
14. KPF_SWAPBACKED page is swap/RAM backed
15. KPF_COMPOUND_HEAD (*)
16. KPF_COMPOUND_TAIL (*)
17. KPF_HUGE hugeTLB pages
18. KPF_UNEVICTABLE page is in the unevictable LRU list
19. KPF_HWPOISON hardware detected corruption
20. KPF_NOPAGE (pseudo flag) no page frame at the address
(*) For compound pages, exporting _both_ head/tail info enables
users to tell where a compound page starts/ends, and its order.
a simple demo of the page-types tool
# ./page-types -h
page-types [options]
-r|--raw Raw mode, for kernel developers
-a|--addr addr-spec Walk a range of pages
-b|--bits bits-spec Walk pages with specified bits
-l|--list Show page details in ranges
-L|--list-each Show page details one by one
-N|--no-summary Don't show summay info
-h|--help Show this usage message
addr-spec:
N one page at offset N (unit: pages)
N+M pages range from N to N+M-1
N,M pages range from N to M-1
N, pages range from N to end
,M pages range from 0 to M
bits-spec:
bit1,bit2 (flags & (bit1|bit2)) != 0
bit1,bit2=bit1 (flags & (bit1|bit2)) == bit1
bit1,~bit2 (flags & (bit1|bit2)) == bit1
=bit1,bit2 flags == (bit1|bit2)
bit-names:
locked error referenced uptodate
dirty lru active slab
writeback reclaim buddy mmap
anonymous swapcache swapbacked compound_head
compound_tail huge unevictable hwpoison
nopage reserved(r) mlocked(r) mappedtodisk(r)
private(r) private_2(r) owner_private(r) arch(r)
uncached(r) readahead(o) slob_free(o) slub_frozen(o)
slub_debug(o)
(r) raw mode bits (o) overloaded bits
# ./page-types
flags page-count MB symbolic-flags long-symbolic-flags
0x0000000000000000 487369 1903 _________________________________
0x0000000000000014 5 0 __R_D____________________________ referenced,dirty
0x0000000000000020 1 0 _____l___________________________ lru
0x0000000000000024 34 0 __R__l___________________________ referenced,lru
0x0000000000000028 3838 14 ___U_l___________________________ uptodate,lru
0x0001000000000028 48 0 ___U_l_______________________I___ uptodate,lru,readahead
0x000000000000002c 6478 25 __RU_l___________________________ referenced,uptodate,lru
0x000100000000002c 47 0 __RU_l_______________________I___ referenced,uptodate,lru,readahead
0x0000000000000040 8344 32 ______A__________________________ active
0x0000000000000060 1 0 _____lA__________________________ lru,active
0x0000000000000068 348 1 ___U_lA__________________________ uptodate,lru,active
0x0001000000000068 12 0 ___U_lA______________________I___ uptodate,lru,active,readahead
0x000000000000006c 988 3 __RU_lA__________________________ referenced,uptodate,lru,active
0x000100000000006c 48 0 __RU_lA______________________I___ referenced,uptodate,lru,active,readahead
0x0000000000004078 1 0 ___UDlA_______b__________________ uptodate,dirty,lru,active,swapbacked
0x000000000000407c 34 0 __RUDlA_______b__________________ referenced,uptodate,dirty,lru,active,swapbacked
0x0000000000000400 503 1 __________B______________________ buddy
0x0000000000000804 1 0 __R________M_____________________ referenced,mmap
0x0000000000000828 1029 4 ___U_l_____M_____________________ uptodate,lru,mmap
0x0001000000000828 43 0 ___U_l_____M_________________I___ uptodate,lru,mmap,readahead
0x000000000000082c 382 1 __RU_l_____M_____________________ referenced,uptodate,lru,mmap
0x000100000000082c 12 0 __RU_l_____M_________________I___ referenced,uptodate,lru,mmap,readahead
0x0000000000000868 192 0 ___U_lA____M_____________________ uptodate,lru,active,mmap
0x0001000000000868 12 0 ___U_lA____M_________________I___ uptodate,lru,active,mmap,readahead
0x000000000000086c 800 3 __RU_lA____M_____________________ referenced,uptodate,lru,active,mmap
0x000100000000086c 31 0 __RU_lA____M_________________I___ referenced,uptodate,lru,active,mmap,readahead
0x0000000000004878 2 0 ___UDlA____M__b__________________ uptodate,dirty,lru,active,mmap,swapbacked
0x0000000000001000 492 1 ____________a____________________ anonymous
0x0000000000005808 4 0 ___U_______Ma_b__________________ uptodate,mmap,anonymous,swapbacked
0x0000000000005868 2839 11 ___U_lA____Ma_b__________________ uptodate,lru,active,mmap,anonymous,swapbacked
0x000000000000586c 30 0 __RU_lA____Ma_b__________________ referenced,uptodate,lru,active,mmap,anonymous,swapbacked
total 513968 2007
# ./page-types -r
flags page-count MB symbolic-flags long-symbolic-flags
0x0000000000000000 468002 1828 _________________________________
0x0000000100000000 19102 74 _____________________r___________ reserved
0x0000000000008000 41 0 _______________H_________________ compound_head
0x0000000000010000 188 0 ________________T________________ compound_tail
0x0000000000008014 1 0 __R_D__________H_________________ referenced,dirty,compound_head
0x0000000000010014 4 0 __R_D___________T________________ referenced,dirty,compound_tail
0x0000000000000020 1 0 _____l___________________________ lru
0x0000000800000024 34 0 __R__l__________________P________ referenced,lru,private
0x0000000000000028 3794 14 ___U_l___________________________ uptodate,lru
0x0001000000000028 46 0 ___U_l_______________________I___ uptodate,lru,readahead
0x0000000400000028 44 0 ___U_l_________________d_________ uptodate,lru,mappedtodisk
0x0001000400000028 2 0 ___U_l_________________d_____I___ uptodate,lru,mappedtodisk,readahead
0x000000000000002c 6434 25 __RU_l___________________________ referenced,uptodate,lru
0x000100000000002c 47 0 __RU_l_______________________I___ referenced,uptodate,lru,readahead
0x000000040000002c 14 0 __RU_l_________________d_________ referenced,uptodate,lru,mappedtodisk
0x000000080000002c 30 0 __RU_l__________________P________ referenced,uptodate,lru,private
0x0000000800000040 8124 31 ______A_________________P________ active,private
0x0000000000000040 219 0 ______A__________________________ active
0x0000000800000060 1 0 _____lA_________________P________ lru,active,private
0x0000000000000068 322 1 ___U_lA__________________________ uptodate,lru,active
0x0001000000000068 12 0 ___U_lA______________________I___ uptodate,lru,active,readahead
0x0000000400000068 13 0 ___U_lA________________d_________ uptodate,lru,active,mappedtodisk
0x0000000800000068 12 0 ___U_lA_________________P________ uptodate,lru,active,private
0x000000000000006c 977 3 __RU_lA__________________________ referenced,uptodate,lru,active
0x000100000000006c 48 0 __RU_lA______________________I___ referenced,uptodate,lru,active,readahead
0x000000040000006c 5 0 __RU_lA________________d_________ referenced,uptodate,lru,active,mappedtodisk
0x000000080000006c 3 0 __RU_lA_________________P________ referenced,uptodate,lru,active,private
0x0000000c0000006c 3 0 __RU_lA________________dP________ referenced,uptodate,lru,active,mappedtodisk,private
0x0000000c00000068 1 0 ___U_lA________________dP________ uptodate,lru,active,mappedtodisk,private
0x0000000000004078 1 0 ___UDlA_______b__________________ uptodate,dirty,lru,active,swapbacked
0x000000000000407c 34 0 __RUDlA_______b__________________ referenced,uptodate,dirty,lru,active,swapbacked
0x0000000000000400 538 2 __________B______________________ buddy
0x0000000000000804 1 0 __R________M_____________________ referenced,mmap
0x0000000000000828 1029 4 ___U_l_____M_____________________ uptodate,lru,mmap
0x0001000000000828 43 0 ___U_l_____M_________________I___ uptodate,lru,mmap,readahead
0x000000000000082c 382 1 __RU_l_____M_____________________ referenced,uptodate,lru,mmap
0x000100000000082c 12 0 __RU_l_____M_________________I___ referenced,uptodate,lru,mmap,readahead
0x0000000000000868 192 0 ___U_lA____M_____________________ uptodate,lru,active,mmap
0x0001000000000868 12 0 ___U_lA____M_________________I___ uptodate,lru,active,mmap,readahead
0x000000000000086c 800 3 __RU_lA____M_____________________ referenced,uptodate,lru,active,mmap
0x000100000000086c 31 0 __RU_lA____M_________________I___ referenced,uptodate,lru,active,mmap,readahead
0x0000000000004878 2 0 ___UDlA____M__b__________________ uptodate,dirty,lru,active,mmap,swapbacked
0x0000000000001000 492 1 ____________a____________________ anonymous
0x0000000000005008 2 0 ___U________a_b__________________ uptodate,anonymous,swapbacked
0x0000000000005808 4 0 ___U_______Ma_b__________________ uptodate,mmap,anonymous,swapbacked
0x000000000000580c 1 0 __RU_______Ma_b__________________ referenced,uptodate,mmap,anonymous,swapbacked
0x0000000000005868 2839 11 ___U_lA____Ma_b__________________ uptodate,lru,active,mmap,anonymous,swapbacked
0x000000000000586c 29 0 __RU_lA____Ma_b__________________ referenced,uptodate,lru,active,mmap,anonymous,swapbacked
total 513968 2007
# ./page-types --raw --list --no-summary --bits reserved
offset count flags
0 15 _____________________r___________
31 4 _____________________r___________
159 97 _____________________r___________
4096 2067 _____________________r___________
6752 2390 _____________________r___________
9355 3 _____________________r___________
9728 14526 _____________________r___________
This patch:
Introduce PageHuge(), which identifies huge/gigantic pages by their
dedicated compound destructor functions.
Also move prep_compound_gigantic_page() to hugetlb.c and make
__free_pages_ok() non-static.
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: Matt Mackall <mpm@selenic.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2009-06-17 05:32:22 +07:00
|
|
|
static void prep_compound_gigantic_page(struct page *page, unsigned long order)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
int nr_pages = 1 << order;
|
|
|
|
struct page *p = page + 1;
|
|
|
|
|
|
|
|
/* we rely on prep_new_huge_page to set the destructor */
|
|
|
|
set_compound_order(page, order);
|
|
|
|
__SetPageHead(page);
|
|
|
|
for (i = 1; i < nr_pages; i++, p = mem_map_next(p, page, i)) {
|
|
|
|
__SetPageTail(p);
|
|
|
|
p->first_page = page;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
int PageHuge(struct page *page)
|
|
|
|
{
|
|
|
|
compound_page_dtor *dtor;
|
|
|
|
|
|
|
|
if (!PageCompound(page))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
page = compound_head(page);
|
|
|
|
dtor = get_compound_page_dtor(page);
|
|
|
|
|
|
|
|
return dtor == free_huge_page;
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
static struct page *alloc_fresh_huge_page_node(struct hstate *h, int nid)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
struct page *page;
|
2007-07-16 13:38:12 +07:00
|
|
|
|
2008-07-24 11:27:47 +07:00
|
|
|
if (h->order >= MAX_ORDER)
|
|
|
|
return NULL;
|
|
|
|
|
2009-06-17 05:31:54 +07:00
|
|
|
page = alloc_pages_exact_node(nid,
|
2008-04-29 14:58:26 +07:00
|
|
|
htlb_alloc_mask|__GFP_COMP|__GFP_THISNODE|
|
|
|
|
__GFP_REPEAT|__GFP_NOWARN,
|
2008-07-24 11:27:41 +07:00
|
|
|
huge_page_order(h));
|
2005-04-17 05:20:36 +07:00
|
|
|
if (page) {
|
2008-04-28 16:13:29 +07:00
|
|
|
if (arch_prepare_hugepage(page)) {
|
2008-08-13 05:08:38 +07:00
|
|
|
__free_pages(page, huge_page_order(h));
|
2008-04-29 04:13:19 +07:00
|
|
|
return NULL;
|
2008-04-28 16:13:29 +07:00
|
|
|
}
|
2008-07-24 11:27:41 +07:00
|
|
|
prep_new_huge_page(h, page, nid);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
hugetlb: fix hugepage allocation with memoryless nodes
Anton found a problem with the hugetlb pool allocation when some nodes have
no memory (http://marc.info/?l=linux-mm&m=118133042025995&w=2). Lee worked
on versions that tried to fix it, but none were accepted. Christoph has
created a set of patches which allow for GFP_THISNODE allocations to fail
if the node has no memory.
Currently, alloc_fresh_huge_page() returns NULL when it is not able to
allocate a huge page on the current node, as specified by its custom
interleave variable. The callers of this function, though, assume that a
failure in alloc_fresh_huge_page() indicates no hugepages can be allocated
on the system period. This might not be the case, for instance, if we have
an uneven NUMA system, and we happen to try to allocate a hugepage on a
node with less memory and fail, while there is still plenty of free memory
on the other nodes.
To correct this, make alloc_fresh_huge_page() search through all online
nodes before deciding no hugepages can be allocated. Add a helper function
for actually allocating the hugepage. Use a new global nid iterator to
control which nid to allocate on.
Note: we expect particular semantics for __GFP_THISNODE, which are now
enforced even for memoryless nodes. That is, there is should be no
fallback to other nodes. Therefore, we rely on the nid passed into
alloc_pages_node() to be the nid the page comes from. If this is
incorrect, accounting will break.
Tested on x86 !NUMA, x86 NUMA, x86_64 NUMA and ppc64 NUMA (with 2
memoryless nodes).
Before on the ppc64 box:
Trying to clear the hugetlb pool
Done. 0 free
Trying to resize the pool to 100
Node 0 HugePages_Free: 25
Node 1 HugePages_Free: 75
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. Initially 100 free
Trying to resize the pool to 200
Node 0 HugePages_Free: 50
Node 1 HugePages_Free: 150
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. 200 free
After:
Trying to clear the hugetlb pool
Done. 0 free
Trying to resize the pool to 100
Node 0 HugePages_Free: 50
Node 1 HugePages_Free: 50
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. Initially 100 free
Trying to resize the pool to 200
Node 0 HugePages_Free: 100
Node 1 HugePages_Free: 100
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. 200 free
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Acked-by: Christoph Lameter <clameter@sgi.com>
Cc: Adam Litke <agl@us.ibm.com>
Cc: David Gibson <hermes@gibson.dropbear.id.au>
Cc: Badari Pulavarty <pbadari@us.ibm.com>
Cc: Ken Chen <kenchen@google.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-16 15:26:24 +07:00
|
|
|
|
|
|
|
return page;
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:45 +07:00
|
|
|
/*
|
|
|
|
* Use a helper variable to find the next node and then
|
|
|
|
* copy it back to hugetlb_next_nid afterwards:
|
|
|
|
* otherwise there's a window in which a racer might
|
2009-06-17 05:31:54 +07:00
|
|
|
* pass invalid nid MAX_NUMNODES to alloc_pages_exact_node.
|
2008-07-24 11:27:45 +07:00
|
|
|
* But we don't need to use a spin_lock here: it really
|
|
|
|
* doesn't matter if occasionally a racer chooses the
|
|
|
|
* same nid as we do. Move nid forward in the mask even
|
|
|
|
* if we just successfully allocated a hugepage so that
|
|
|
|
* the next caller gets hugepages on the next node.
|
|
|
|
*/
|
|
|
|
static int hstate_next_node(struct hstate *h)
|
|
|
|
{
|
|
|
|
int next_nid;
|
|
|
|
next_nid = next_node(h->hugetlb_next_nid, node_online_map);
|
|
|
|
if (next_nid == MAX_NUMNODES)
|
|
|
|
next_nid = first_node(node_online_map);
|
|
|
|
h->hugetlb_next_nid = next_nid;
|
|
|
|
return next_nid;
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
static int alloc_fresh_huge_page(struct hstate *h)
|
hugetlb: fix hugepage allocation with memoryless nodes
Anton found a problem with the hugetlb pool allocation when some nodes have
no memory (http://marc.info/?l=linux-mm&m=118133042025995&w=2). Lee worked
on versions that tried to fix it, but none were accepted. Christoph has
created a set of patches which allow for GFP_THISNODE allocations to fail
if the node has no memory.
Currently, alloc_fresh_huge_page() returns NULL when it is not able to
allocate a huge page on the current node, as specified by its custom
interleave variable. The callers of this function, though, assume that a
failure in alloc_fresh_huge_page() indicates no hugepages can be allocated
on the system period. This might not be the case, for instance, if we have
an uneven NUMA system, and we happen to try to allocate a hugepage on a
node with less memory and fail, while there is still plenty of free memory
on the other nodes.
To correct this, make alloc_fresh_huge_page() search through all online
nodes before deciding no hugepages can be allocated. Add a helper function
for actually allocating the hugepage. Use a new global nid iterator to
control which nid to allocate on.
Note: we expect particular semantics for __GFP_THISNODE, which are now
enforced even for memoryless nodes. That is, there is should be no
fallback to other nodes. Therefore, we rely on the nid passed into
alloc_pages_node() to be the nid the page comes from. If this is
incorrect, accounting will break.
Tested on x86 !NUMA, x86 NUMA, x86_64 NUMA and ppc64 NUMA (with 2
memoryless nodes).
Before on the ppc64 box:
Trying to clear the hugetlb pool
Done. 0 free
Trying to resize the pool to 100
Node 0 HugePages_Free: 25
Node 1 HugePages_Free: 75
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. Initially 100 free
Trying to resize the pool to 200
Node 0 HugePages_Free: 50
Node 1 HugePages_Free: 150
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. 200 free
After:
Trying to clear the hugetlb pool
Done. 0 free
Trying to resize the pool to 100
Node 0 HugePages_Free: 50
Node 1 HugePages_Free: 50
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. Initially 100 free
Trying to resize the pool to 200
Node 0 HugePages_Free: 100
Node 1 HugePages_Free: 100
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. 200 free
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Acked-by: Christoph Lameter <clameter@sgi.com>
Cc: Adam Litke <agl@us.ibm.com>
Cc: David Gibson <hermes@gibson.dropbear.id.au>
Cc: Badari Pulavarty <pbadari@us.ibm.com>
Cc: Ken Chen <kenchen@google.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-16 15:26:24 +07:00
|
|
|
{
|
|
|
|
struct page *page;
|
|
|
|
int start_nid;
|
|
|
|
int next_nid;
|
|
|
|
int ret = 0;
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
start_nid = h->hugetlb_next_nid;
|
hugetlb: fix hugepage allocation with memoryless nodes
Anton found a problem with the hugetlb pool allocation when some nodes have
no memory (http://marc.info/?l=linux-mm&m=118133042025995&w=2). Lee worked
on versions that tried to fix it, but none were accepted. Christoph has
created a set of patches which allow for GFP_THISNODE allocations to fail
if the node has no memory.
Currently, alloc_fresh_huge_page() returns NULL when it is not able to
allocate a huge page on the current node, as specified by its custom
interleave variable. The callers of this function, though, assume that a
failure in alloc_fresh_huge_page() indicates no hugepages can be allocated
on the system period. This might not be the case, for instance, if we have
an uneven NUMA system, and we happen to try to allocate a hugepage on a
node with less memory and fail, while there is still plenty of free memory
on the other nodes.
To correct this, make alloc_fresh_huge_page() search through all online
nodes before deciding no hugepages can be allocated. Add a helper function
for actually allocating the hugepage. Use a new global nid iterator to
control which nid to allocate on.
Note: we expect particular semantics for __GFP_THISNODE, which are now
enforced even for memoryless nodes. That is, there is should be no
fallback to other nodes. Therefore, we rely on the nid passed into
alloc_pages_node() to be the nid the page comes from. If this is
incorrect, accounting will break.
Tested on x86 !NUMA, x86 NUMA, x86_64 NUMA and ppc64 NUMA (with 2
memoryless nodes).
Before on the ppc64 box:
Trying to clear the hugetlb pool
Done. 0 free
Trying to resize the pool to 100
Node 0 HugePages_Free: 25
Node 1 HugePages_Free: 75
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. Initially 100 free
Trying to resize the pool to 200
Node 0 HugePages_Free: 50
Node 1 HugePages_Free: 150
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. 200 free
After:
Trying to clear the hugetlb pool
Done. 0 free
Trying to resize the pool to 100
Node 0 HugePages_Free: 50
Node 1 HugePages_Free: 50
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. Initially 100 free
Trying to resize the pool to 200
Node 0 HugePages_Free: 100
Node 1 HugePages_Free: 100
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. 200 free
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Acked-by: Christoph Lameter <clameter@sgi.com>
Cc: Adam Litke <agl@us.ibm.com>
Cc: David Gibson <hermes@gibson.dropbear.id.au>
Cc: Badari Pulavarty <pbadari@us.ibm.com>
Cc: Ken Chen <kenchen@google.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-16 15:26:24 +07:00
|
|
|
|
|
|
|
do {
|
2008-07-24 11:27:41 +07:00
|
|
|
page = alloc_fresh_huge_page_node(h, h->hugetlb_next_nid);
|
hugetlb: fix hugepage allocation with memoryless nodes
Anton found a problem with the hugetlb pool allocation when some nodes have
no memory (http://marc.info/?l=linux-mm&m=118133042025995&w=2). Lee worked
on versions that tried to fix it, but none were accepted. Christoph has
created a set of patches which allow for GFP_THISNODE allocations to fail
if the node has no memory.
Currently, alloc_fresh_huge_page() returns NULL when it is not able to
allocate a huge page on the current node, as specified by its custom
interleave variable. The callers of this function, though, assume that a
failure in alloc_fresh_huge_page() indicates no hugepages can be allocated
on the system period. This might not be the case, for instance, if we have
an uneven NUMA system, and we happen to try to allocate a hugepage on a
node with less memory and fail, while there is still plenty of free memory
on the other nodes.
To correct this, make alloc_fresh_huge_page() search through all online
nodes before deciding no hugepages can be allocated. Add a helper function
for actually allocating the hugepage. Use a new global nid iterator to
control which nid to allocate on.
Note: we expect particular semantics for __GFP_THISNODE, which are now
enforced even for memoryless nodes. That is, there is should be no
fallback to other nodes. Therefore, we rely on the nid passed into
alloc_pages_node() to be the nid the page comes from. If this is
incorrect, accounting will break.
Tested on x86 !NUMA, x86 NUMA, x86_64 NUMA and ppc64 NUMA (with 2
memoryless nodes).
Before on the ppc64 box:
Trying to clear the hugetlb pool
Done. 0 free
Trying to resize the pool to 100
Node 0 HugePages_Free: 25
Node 1 HugePages_Free: 75
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. Initially 100 free
Trying to resize the pool to 200
Node 0 HugePages_Free: 50
Node 1 HugePages_Free: 150
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. 200 free
After:
Trying to clear the hugetlb pool
Done. 0 free
Trying to resize the pool to 100
Node 0 HugePages_Free: 50
Node 1 HugePages_Free: 50
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. Initially 100 free
Trying to resize the pool to 200
Node 0 HugePages_Free: 100
Node 1 HugePages_Free: 100
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. 200 free
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Acked-by: Christoph Lameter <clameter@sgi.com>
Cc: Adam Litke <agl@us.ibm.com>
Cc: David Gibson <hermes@gibson.dropbear.id.au>
Cc: Badari Pulavarty <pbadari@us.ibm.com>
Cc: Ken Chen <kenchen@google.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-16 15:26:24 +07:00
|
|
|
if (page)
|
|
|
|
ret = 1;
|
2008-07-24 11:27:45 +07:00
|
|
|
next_nid = hstate_next_node(h);
|
2008-07-24 11:27:41 +07:00
|
|
|
} while (!page && h->hugetlb_next_nid != start_nid);
|
hugetlb: fix hugepage allocation with memoryless nodes
Anton found a problem with the hugetlb pool allocation when some nodes have
no memory (http://marc.info/?l=linux-mm&m=118133042025995&w=2). Lee worked
on versions that tried to fix it, but none were accepted. Christoph has
created a set of patches which allow for GFP_THISNODE allocations to fail
if the node has no memory.
Currently, alloc_fresh_huge_page() returns NULL when it is not able to
allocate a huge page on the current node, as specified by its custom
interleave variable. The callers of this function, though, assume that a
failure in alloc_fresh_huge_page() indicates no hugepages can be allocated
on the system period. This might not be the case, for instance, if we have
an uneven NUMA system, and we happen to try to allocate a hugepage on a
node with less memory and fail, while there is still plenty of free memory
on the other nodes.
To correct this, make alloc_fresh_huge_page() search through all online
nodes before deciding no hugepages can be allocated. Add a helper function
for actually allocating the hugepage. Use a new global nid iterator to
control which nid to allocate on.
Note: we expect particular semantics for __GFP_THISNODE, which are now
enforced even for memoryless nodes. That is, there is should be no
fallback to other nodes. Therefore, we rely on the nid passed into
alloc_pages_node() to be the nid the page comes from. If this is
incorrect, accounting will break.
Tested on x86 !NUMA, x86 NUMA, x86_64 NUMA and ppc64 NUMA (with 2
memoryless nodes).
Before on the ppc64 box:
Trying to clear the hugetlb pool
Done. 0 free
Trying to resize the pool to 100
Node 0 HugePages_Free: 25
Node 1 HugePages_Free: 75
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. Initially 100 free
Trying to resize the pool to 200
Node 0 HugePages_Free: 50
Node 1 HugePages_Free: 150
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. 200 free
After:
Trying to clear the hugetlb pool
Done. 0 free
Trying to resize the pool to 100
Node 0 HugePages_Free: 50
Node 1 HugePages_Free: 50
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. Initially 100 free
Trying to resize the pool to 200
Node 0 HugePages_Free: 100
Node 1 HugePages_Free: 100
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. 200 free
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Acked-by: Christoph Lameter <clameter@sgi.com>
Cc: Adam Litke <agl@us.ibm.com>
Cc: David Gibson <hermes@gibson.dropbear.id.au>
Cc: Badari Pulavarty <pbadari@us.ibm.com>
Cc: Ken Chen <kenchen@google.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-16 15:26:24 +07:00
|
|
|
|
2008-04-28 16:13:06 +07:00
|
|
|
if (ret)
|
|
|
|
count_vm_event(HTLB_BUDDY_PGALLOC);
|
|
|
|
else
|
|
|
|
count_vm_event(HTLB_BUDDY_PGALLOC_FAIL);
|
|
|
|
|
hugetlb: fix hugepage allocation with memoryless nodes
Anton found a problem with the hugetlb pool allocation when some nodes have
no memory (http://marc.info/?l=linux-mm&m=118133042025995&w=2). Lee worked
on versions that tried to fix it, but none were accepted. Christoph has
created a set of patches which allow for GFP_THISNODE allocations to fail
if the node has no memory.
Currently, alloc_fresh_huge_page() returns NULL when it is not able to
allocate a huge page on the current node, as specified by its custom
interleave variable. The callers of this function, though, assume that a
failure in alloc_fresh_huge_page() indicates no hugepages can be allocated
on the system period. This might not be the case, for instance, if we have
an uneven NUMA system, and we happen to try to allocate a hugepage on a
node with less memory and fail, while there is still plenty of free memory
on the other nodes.
To correct this, make alloc_fresh_huge_page() search through all online
nodes before deciding no hugepages can be allocated. Add a helper function
for actually allocating the hugepage. Use a new global nid iterator to
control which nid to allocate on.
Note: we expect particular semantics for __GFP_THISNODE, which are now
enforced even for memoryless nodes. That is, there is should be no
fallback to other nodes. Therefore, we rely on the nid passed into
alloc_pages_node() to be the nid the page comes from. If this is
incorrect, accounting will break.
Tested on x86 !NUMA, x86 NUMA, x86_64 NUMA and ppc64 NUMA (with 2
memoryless nodes).
Before on the ppc64 box:
Trying to clear the hugetlb pool
Done. 0 free
Trying to resize the pool to 100
Node 0 HugePages_Free: 25
Node 1 HugePages_Free: 75
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. Initially 100 free
Trying to resize the pool to 200
Node 0 HugePages_Free: 50
Node 1 HugePages_Free: 150
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. 200 free
After:
Trying to clear the hugetlb pool
Done. 0 free
Trying to resize the pool to 100
Node 0 HugePages_Free: 50
Node 1 HugePages_Free: 50
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. Initially 100 free
Trying to resize the pool to 200
Node 0 HugePages_Free: 100
Node 1 HugePages_Free: 100
Node 2 HugePages_Free: 0
Node 3 HugePages_Free: 0
Done. 200 free
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Acked-by: Christoph Lameter <clameter@sgi.com>
Cc: Adam Litke <agl@us.ibm.com>
Cc: David Gibson <hermes@gibson.dropbear.id.au>
Cc: Badari Pulavarty <pbadari@us.ibm.com>
Cc: Ken Chen <kenchen@google.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
Cc: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-10-16 15:26:24 +07:00
|
|
|
return ret;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
static struct page *alloc_buddy_huge_page(struct hstate *h,
|
|
|
|
struct vm_area_struct *vma, unsigned long address)
|
2007-10-16 15:26:18 +07:00
|
|
|
{
|
|
|
|
struct page *page;
|
hugetlb: introduce nr_overcommit_hugepages sysctl
hugetlb: introduce nr_overcommit_hugepages sysctl
While examining the code to support /proc/sys/vm/hugetlb_dynamic_pool, I
became convinced that having a boolean sysctl was insufficient:
1) To support per-node control of hugepages, I have previously submitted
patches to add a sysfs attribute related to nr_hugepages. However, with
a boolean global value and per-mount quota enforcement constraining the
dynamic pool, adding corresponding control of the dynamic pool on a
per-node basis seems inconsistent to me.
2) Administration of the hugetlb dynamic pool with multiple hugetlbfs
mount points is, arguably, more arduous than it needs to be. Each quota
would need to be set separately, and the sum would need to be monitored.
To ease the administration, and to help make the way for per-node
control of the static & dynamic hugepage pool, I added a separate
sysctl, nr_overcommit_hugepages. This value serves as a high watermark
for the overall hugepage pool, while nr_hugepages serves as a low
watermark. The boolean sysctl can then be removed, as the condition
nr_overcommit_hugepages > 0
indicates the same administrative setting as
hugetlb_dynamic_pool == 1
Quotas still serve as local enforcement of the size of the pool on a
per-mount basis.
A few caveats:
1) There is a race whereby the global surplus huge page counter is
incremented before a hugepage has allocated. Another process could then
try grow the pool, and fail to convert a surplus huge page to a normal
huge page and instead allocate a fresh huge page. I believe this is
benign, as no memory is leaked (the actual pages are still tracked
correctly) and the counters won't go out of sync.
2) Shrinking the static pool while a surplus is in effect will allow the
number of surplus huge pages to exceed the overcommit value. As long as
this condition holds, however, no more surplus huge pages will be
allowed on the system until one of the two sysctls are increased
sufficiently, or the surplus huge pages go out of use and are freed.
Successfully tested on x86_64 with the current libhugetlbfs snapshot,
modified to use the new sysctl.
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Dave Hansen <haveblue@us.ibm.com>
Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-12-18 07:20:12 +07:00
|
|
|
unsigned int nid;
|
2007-10-16 15:26:18 +07:00
|
|
|
|
2008-07-24 11:27:47 +07:00
|
|
|
if (h->order >= MAX_ORDER)
|
|
|
|
return NULL;
|
|
|
|
|
hugetlb: introduce nr_overcommit_hugepages sysctl
hugetlb: introduce nr_overcommit_hugepages sysctl
While examining the code to support /proc/sys/vm/hugetlb_dynamic_pool, I
became convinced that having a boolean sysctl was insufficient:
1) To support per-node control of hugepages, I have previously submitted
patches to add a sysfs attribute related to nr_hugepages. However, with
a boolean global value and per-mount quota enforcement constraining the
dynamic pool, adding corresponding control of the dynamic pool on a
per-node basis seems inconsistent to me.
2) Administration of the hugetlb dynamic pool with multiple hugetlbfs
mount points is, arguably, more arduous than it needs to be. Each quota
would need to be set separately, and the sum would need to be monitored.
To ease the administration, and to help make the way for per-node
control of the static & dynamic hugepage pool, I added a separate
sysctl, nr_overcommit_hugepages. This value serves as a high watermark
for the overall hugepage pool, while nr_hugepages serves as a low
watermark. The boolean sysctl can then be removed, as the condition
nr_overcommit_hugepages > 0
indicates the same administrative setting as
hugetlb_dynamic_pool == 1
Quotas still serve as local enforcement of the size of the pool on a
per-mount basis.
A few caveats:
1) There is a race whereby the global surplus huge page counter is
incremented before a hugepage has allocated. Another process could then
try grow the pool, and fail to convert a surplus huge page to a normal
huge page and instead allocate a fresh huge page. I believe this is
benign, as no memory is leaked (the actual pages are still tracked
correctly) and the counters won't go out of sync.
2) Shrinking the static pool while a surplus is in effect will allow the
number of surplus huge pages to exceed the overcommit value. As long as
this condition holds, however, no more surplus huge pages will be
allowed on the system until one of the two sysctls are increased
sufficiently, or the surplus huge pages go out of use and are freed.
Successfully tested on x86_64 with the current libhugetlbfs snapshot,
modified to use the new sysctl.
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Dave Hansen <haveblue@us.ibm.com>
Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-12-18 07:20:12 +07:00
|
|
|
/*
|
|
|
|
* Assume we will successfully allocate the surplus page to
|
|
|
|
* prevent racing processes from causing the surplus to exceed
|
|
|
|
* overcommit
|
|
|
|
*
|
|
|
|
* This however introduces a different race, where a process B
|
|
|
|
* tries to grow the static hugepage pool while alloc_pages() is
|
|
|
|
* called by process A. B will only examine the per-node
|
|
|
|
* counters in determining if surplus huge pages can be
|
|
|
|
* converted to normal huge pages in adjust_pool_surplus(). A
|
|
|
|
* won't be able to increment the per-node counter, until the
|
|
|
|
* lock is dropped by B, but B doesn't drop hugetlb_lock until
|
|
|
|
* no more huge pages can be converted from surplus to normal
|
|
|
|
* state (and doesn't try to convert again). Thus, we have a
|
|
|
|
* case where a surplus huge page exists, the pool is grown, and
|
|
|
|
* the surplus huge page still exists after, even though it
|
|
|
|
* should just have been converted to a normal huge page. This
|
|
|
|
* does not leak memory, though, as the hugepage will be freed
|
|
|
|
* once it is out of use. It also does not allow the counters to
|
|
|
|
* go out of whack in adjust_pool_surplus() as we don't modify
|
|
|
|
* the node values until we've gotten the hugepage and only the
|
|
|
|
* per-node value is checked there.
|
|
|
|
*/
|
|
|
|
spin_lock(&hugetlb_lock);
|
2008-07-24 11:27:41 +07:00
|
|
|
if (h->surplus_huge_pages >= h->nr_overcommit_huge_pages) {
|
hugetlb: introduce nr_overcommit_hugepages sysctl
hugetlb: introduce nr_overcommit_hugepages sysctl
While examining the code to support /proc/sys/vm/hugetlb_dynamic_pool, I
became convinced that having a boolean sysctl was insufficient:
1) To support per-node control of hugepages, I have previously submitted
patches to add a sysfs attribute related to nr_hugepages. However, with
a boolean global value and per-mount quota enforcement constraining the
dynamic pool, adding corresponding control of the dynamic pool on a
per-node basis seems inconsistent to me.
2) Administration of the hugetlb dynamic pool with multiple hugetlbfs
mount points is, arguably, more arduous than it needs to be. Each quota
would need to be set separately, and the sum would need to be monitored.
To ease the administration, and to help make the way for per-node
control of the static & dynamic hugepage pool, I added a separate
sysctl, nr_overcommit_hugepages. This value serves as a high watermark
for the overall hugepage pool, while nr_hugepages serves as a low
watermark. The boolean sysctl can then be removed, as the condition
nr_overcommit_hugepages > 0
indicates the same administrative setting as
hugetlb_dynamic_pool == 1
Quotas still serve as local enforcement of the size of the pool on a
per-mount basis.
A few caveats:
1) There is a race whereby the global surplus huge page counter is
incremented before a hugepage has allocated. Another process could then
try grow the pool, and fail to convert a surplus huge page to a normal
huge page and instead allocate a fresh huge page. I believe this is
benign, as no memory is leaked (the actual pages are still tracked
correctly) and the counters won't go out of sync.
2) Shrinking the static pool while a surplus is in effect will allow the
number of surplus huge pages to exceed the overcommit value. As long as
this condition holds, however, no more surplus huge pages will be
allowed on the system until one of the two sysctls are increased
sufficiently, or the surplus huge pages go out of use and are freed.
Successfully tested on x86_64 with the current libhugetlbfs snapshot,
modified to use the new sysctl.
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Dave Hansen <haveblue@us.ibm.com>
Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-12-18 07:20:12 +07:00
|
|
|
spin_unlock(&hugetlb_lock);
|
|
|
|
return NULL;
|
|
|
|
} else {
|
2008-07-24 11:27:41 +07:00
|
|
|
h->nr_huge_pages++;
|
|
|
|
h->surplus_huge_pages++;
|
hugetlb: introduce nr_overcommit_hugepages sysctl
hugetlb: introduce nr_overcommit_hugepages sysctl
While examining the code to support /proc/sys/vm/hugetlb_dynamic_pool, I
became convinced that having a boolean sysctl was insufficient:
1) To support per-node control of hugepages, I have previously submitted
patches to add a sysfs attribute related to nr_hugepages. However, with
a boolean global value and per-mount quota enforcement constraining the
dynamic pool, adding corresponding control of the dynamic pool on a
per-node basis seems inconsistent to me.
2) Administration of the hugetlb dynamic pool with multiple hugetlbfs
mount points is, arguably, more arduous than it needs to be. Each quota
would need to be set separately, and the sum would need to be monitored.
To ease the administration, and to help make the way for per-node
control of the static & dynamic hugepage pool, I added a separate
sysctl, nr_overcommit_hugepages. This value serves as a high watermark
for the overall hugepage pool, while nr_hugepages serves as a low
watermark. The boolean sysctl can then be removed, as the condition
nr_overcommit_hugepages > 0
indicates the same administrative setting as
hugetlb_dynamic_pool == 1
Quotas still serve as local enforcement of the size of the pool on a
per-mount basis.
A few caveats:
1) There is a race whereby the global surplus huge page counter is
incremented before a hugepage has allocated. Another process could then
try grow the pool, and fail to convert a surplus huge page to a normal
huge page and instead allocate a fresh huge page. I believe this is
benign, as no memory is leaked (the actual pages are still tracked
correctly) and the counters won't go out of sync.
2) Shrinking the static pool while a surplus is in effect will allow the
number of surplus huge pages to exceed the overcommit value. As long as
this condition holds, however, no more surplus huge pages will be
allowed on the system until one of the two sysctls are increased
sufficiently, or the surplus huge pages go out of use and are freed.
Successfully tested on x86_64 with the current libhugetlbfs snapshot,
modified to use the new sysctl.
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Dave Hansen <haveblue@us.ibm.com>
Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-12-18 07:20:12 +07:00
|
|
|
}
|
|
|
|
spin_unlock(&hugetlb_lock);
|
|
|
|
|
2008-04-29 14:58:26 +07:00
|
|
|
page = alloc_pages(htlb_alloc_mask|__GFP_COMP|
|
|
|
|
__GFP_REPEAT|__GFP_NOWARN,
|
2008-07-24 11:27:41 +07:00
|
|
|
huge_page_order(h));
|
hugetlb: introduce nr_overcommit_hugepages sysctl
hugetlb: introduce nr_overcommit_hugepages sysctl
While examining the code to support /proc/sys/vm/hugetlb_dynamic_pool, I
became convinced that having a boolean sysctl was insufficient:
1) To support per-node control of hugepages, I have previously submitted
patches to add a sysfs attribute related to nr_hugepages. However, with
a boolean global value and per-mount quota enforcement constraining the
dynamic pool, adding corresponding control of the dynamic pool on a
per-node basis seems inconsistent to me.
2) Administration of the hugetlb dynamic pool with multiple hugetlbfs
mount points is, arguably, more arduous than it needs to be. Each quota
would need to be set separately, and the sum would need to be monitored.
To ease the administration, and to help make the way for per-node
control of the static & dynamic hugepage pool, I added a separate
sysctl, nr_overcommit_hugepages. This value serves as a high watermark
for the overall hugepage pool, while nr_hugepages serves as a low
watermark. The boolean sysctl can then be removed, as the condition
nr_overcommit_hugepages > 0
indicates the same administrative setting as
hugetlb_dynamic_pool == 1
Quotas still serve as local enforcement of the size of the pool on a
per-mount basis.
A few caveats:
1) There is a race whereby the global surplus huge page counter is
incremented before a hugepage has allocated. Another process could then
try grow the pool, and fail to convert a surplus huge page to a normal
huge page and instead allocate a fresh huge page. I believe this is
benign, as no memory is leaked (the actual pages are still tracked
correctly) and the counters won't go out of sync.
2) Shrinking the static pool while a surplus is in effect will allow the
number of surplus huge pages to exceed the overcommit value. As long as
this condition holds, however, no more surplus huge pages will be
allowed on the system until one of the two sysctls are increased
sufficiently, or the surplus huge pages go out of use and are freed.
Successfully tested on x86_64 with the current libhugetlbfs snapshot,
modified to use the new sysctl.
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Dave Hansen <haveblue@us.ibm.com>
Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-12-18 07:20:12 +07:00
|
|
|
|
2008-08-13 05:08:38 +07:00
|
|
|
if (page && arch_prepare_hugepage(page)) {
|
|
|
|
__free_pages(page, huge_page_order(h));
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
hugetlb: introduce nr_overcommit_hugepages sysctl
hugetlb: introduce nr_overcommit_hugepages sysctl
While examining the code to support /proc/sys/vm/hugetlb_dynamic_pool, I
became convinced that having a boolean sysctl was insufficient:
1) To support per-node control of hugepages, I have previously submitted
patches to add a sysfs attribute related to nr_hugepages. However, with
a boolean global value and per-mount quota enforcement constraining the
dynamic pool, adding corresponding control of the dynamic pool on a
per-node basis seems inconsistent to me.
2) Administration of the hugetlb dynamic pool with multiple hugetlbfs
mount points is, arguably, more arduous than it needs to be. Each quota
would need to be set separately, and the sum would need to be monitored.
To ease the administration, and to help make the way for per-node
control of the static & dynamic hugepage pool, I added a separate
sysctl, nr_overcommit_hugepages. This value serves as a high watermark
for the overall hugepage pool, while nr_hugepages serves as a low
watermark. The boolean sysctl can then be removed, as the condition
nr_overcommit_hugepages > 0
indicates the same administrative setting as
hugetlb_dynamic_pool == 1
Quotas still serve as local enforcement of the size of the pool on a
per-mount basis.
A few caveats:
1) There is a race whereby the global surplus huge page counter is
incremented before a hugepage has allocated. Another process could then
try grow the pool, and fail to convert a surplus huge page to a normal
huge page and instead allocate a fresh huge page. I believe this is
benign, as no memory is leaked (the actual pages are still tracked
correctly) and the counters won't go out of sync.
2) Shrinking the static pool while a surplus is in effect will allow the
number of surplus huge pages to exceed the overcommit value. As long as
this condition holds, however, no more surplus huge pages will be
allowed on the system until one of the two sysctls are increased
sufficiently, or the surplus huge pages go out of use and are freed.
Successfully tested on x86_64 with the current libhugetlbfs snapshot,
modified to use the new sysctl.
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Dave Hansen <haveblue@us.ibm.com>
Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-12-18 07:20:12 +07:00
|
|
|
spin_lock(&hugetlb_lock);
|
2007-10-16 15:26:18 +07:00
|
|
|
if (page) {
|
2008-03-11 01:43:50 +07:00
|
|
|
/*
|
|
|
|
* This page is now managed by the hugetlb allocator and has
|
|
|
|
* no users -- drop the buddy allocator's reference.
|
|
|
|
*/
|
|
|
|
put_page_testzero(page);
|
|
|
|
VM_BUG_ON(page_count(page));
|
hugetlb: introduce nr_overcommit_hugepages sysctl
hugetlb: introduce nr_overcommit_hugepages sysctl
While examining the code to support /proc/sys/vm/hugetlb_dynamic_pool, I
became convinced that having a boolean sysctl was insufficient:
1) To support per-node control of hugepages, I have previously submitted
patches to add a sysfs attribute related to nr_hugepages. However, with
a boolean global value and per-mount quota enforcement constraining the
dynamic pool, adding corresponding control of the dynamic pool on a
per-node basis seems inconsistent to me.
2) Administration of the hugetlb dynamic pool with multiple hugetlbfs
mount points is, arguably, more arduous than it needs to be. Each quota
would need to be set separately, and the sum would need to be monitored.
To ease the administration, and to help make the way for per-node
control of the static & dynamic hugepage pool, I added a separate
sysctl, nr_overcommit_hugepages. This value serves as a high watermark
for the overall hugepage pool, while nr_hugepages serves as a low
watermark. The boolean sysctl can then be removed, as the condition
nr_overcommit_hugepages > 0
indicates the same administrative setting as
hugetlb_dynamic_pool == 1
Quotas still serve as local enforcement of the size of the pool on a
per-mount basis.
A few caveats:
1) There is a race whereby the global surplus huge page counter is
incremented before a hugepage has allocated. Another process could then
try grow the pool, and fail to convert a surplus huge page to a normal
huge page and instead allocate a fresh huge page. I believe this is
benign, as no memory is leaked (the actual pages are still tracked
correctly) and the counters won't go out of sync.
2) Shrinking the static pool while a surplus is in effect will allow the
number of surplus huge pages to exceed the overcommit value. As long as
this condition holds, however, no more surplus huge pages will be
allowed on the system until one of the two sysctls are increased
sufficiently, or the surplus huge pages go out of use and are freed.
Successfully tested on x86_64 with the current libhugetlbfs snapshot,
modified to use the new sysctl.
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Dave Hansen <haveblue@us.ibm.com>
Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-12-18 07:20:12 +07:00
|
|
|
nid = page_to_nid(page);
|
2007-10-16 15:26:18 +07:00
|
|
|
set_compound_page_dtor(page, free_huge_page);
|
hugetlb: introduce nr_overcommit_hugepages sysctl
hugetlb: introduce nr_overcommit_hugepages sysctl
While examining the code to support /proc/sys/vm/hugetlb_dynamic_pool, I
became convinced that having a boolean sysctl was insufficient:
1) To support per-node control of hugepages, I have previously submitted
patches to add a sysfs attribute related to nr_hugepages. However, with
a boolean global value and per-mount quota enforcement constraining the
dynamic pool, adding corresponding control of the dynamic pool on a
per-node basis seems inconsistent to me.
2) Administration of the hugetlb dynamic pool with multiple hugetlbfs
mount points is, arguably, more arduous than it needs to be. Each quota
would need to be set separately, and the sum would need to be monitored.
To ease the administration, and to help make the way for per-node
control of the static & dynamic hugepage pool, I added a separate
sysctl, nr_overcommit_hugepages. This value serves as a high watermark
for the overall hugepage pool, while nr_hugepages serves as a low
watermark. The boolean sysctl can then be removed, as the condition
nr_overcommit_hugepages > 0
indicates the same administrative setting as
hugetlb_dynamic_pool == 1
Quotas still serve as local enforcement of the size of the pool on a
per-mount basis.
A few caveats:
1) There is a race whereby the global surplus huge page counter is
incremented before a hugepage has allocated. Another process could then
try grow the pool, and fail to convert a surplus huge page to a normal
huge page and instead allocate a fresh huge page. I believe this is
benign, as no memory is leaked (the actual pages are still tracked
correctly) and the counters won't go out of sync.
2) Shrinking the static pool while a surplus is in effect will allow the
number of surplus huge pages to exceed the overcommit value. As long as
this condition holds, however, no more surplus huge pages will be
allowed on the system until one of the two sysctls are increased
sufficiently, or the surplus huge pages go out of use and are freed.
Successfully tested on x86_64 with the current libhugetlbfs snapshot,
modified to use the new sysctl.
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Dave Hansen <haveblue@us.ibm.com>
Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-12-18 07:20:12 +07:00
|
|
|
/*
|
|
|
|
* We incremented the global counters already
|
|
|
|
*/
|
2008-07-24 11:27:41 +07:00
|
|
|
h->nr_huge_pages_node[nid]++;
|
|
|
|
h->surplus_huge_pages_node[nid]++;
|
2008-04-28 16:13:06 +07:00
|
|
|
__count_vm_event(HTLB_BUDDY_PGALLOC);
|
hugetlb: introduce nr_overcommit_hugepages sysctl
hugetlb: introduce nr_overcommit_hugepages sysctl
While examining the code to support /proc/sys/vm/hugetlb_dynamic_pool, I
became convinced that having a boolean sysctl was insufficient:
1) To support per-node control of hugepages, I have previously submitted
patches to add a sysfs attribute related to nr_hugepages. However, with
a boolean global value and per-mount quota enforcement constraining the
dynamic pool, adding corresponding control of the dynamic pool on a
per-node basis seems inconsistent to me.
2) Administration of the hugetlb dynamic pool with multiple hugetlbfs
mount points is, arguably, more arduous than it needs to be. Each quota
would need to be set separately, and the sum would need to be monitored.
To ease the administration, and to help make the way for per-node
control of the static & dynamic hugepage pool, I added a separate
sysctl, nr_overcommit_hugepages. This value serves as a high watermark
for the overall hugepage pool, while nr_hugepages serves as a low
watermark. The boolean sysctl can then be removed, as the condition
nr_overcommit_hugepages > 0
indicates the same administrative setting as
hugetlb_dynamic_pool == 1
Quotas still serve as local enforcement of the size of the pool on a
per-mount basis.
A few caveats:
1) There is a race whereby the global surplus huge page counter is
incremented before a hugepage has allocated. Another process could then
try grow the pool, and fail to convert a surplus huge page to a normal
huge page and instead allocate a fresh huge page. I believe this is
benign, as no memory is leaked (the actual pages are still tracked
correctly) and the counters won't go out of sync.
2) Shrinking the static pool while a surplus is in effect will allow the
number of surplus huge pages to exceed the overcommit value. As long as
this condition holds, however, no more surplus huge pages will be
allowed on the system until one of the two sysctls are increased
sufficiently, or the surplus huge pages go out of use and are freed.
Successfully tested on x86_64 with the current libhugetlbfs snapshot,
modified to use the new sysctl.
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Dave Hansen <haveblue@us.ibm.com>
Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-12-18 07:20:12 +07:00
|
|
|
} else {
|
2008-07-24 11:27:41 +07:00
|
|
|
h->nr_huge_pages--;
|
|
|
|
h->surplus_huge_pages--;
|
2008-04-28 16:13:06 +07:00
|
|
|
__count_vm_event(HTLB_BUDDY_PGALLOC_FAIL);
|
2007-10-16 15:26:18 +07:00
|
|
|
}
|
hugetlb: introduce nr_overcommit_hugepages sysctl
hugetlb: introduce nr_overcommit_hugepages sysctl
While examining the code to support /proc/sys/vm/hugetlb_dynamic_pool, I
became convinced that having a boolean sysctl was insufficient:
1) To support per-node control of hugepages, I have previously submitted
patches to add a sysfs attribute related to nr_hugepages. However, with
a boolean global value and per-mount quota enforcement constraining the
dynamic pool, adding corresponding control of the dynamic pool on a
per-node basis seems inconsistent to me.
2) Administration of the hugetlb dynamic pool with multiple hugetlbfs
mount points is, arguably, more arduous than it needs to be. Each quota
would need to be set separately, and the sum would need to be monitored.
To ease the administration, and to help make the way for per-node
control of the static & dynamic hugepage pool, I added a separate
sysctl, nr_overcommit_hugepages. This value serves as a high watermark
for the overall hugepage pool, while nr_hugepages serves as a low
watermark. The boolean sysctl can then be removed, as the condition
nr_overcommit_hugepages > 0
indicates the same administrative setting as
hugetlb_dynamic_pool == 1
Quotas still serve as local enforcement of the size of the pool on a
per-mount basis.
A few caveats:
1) There is a race whereby the global surplus huge page counter is
incremented before a hugepage has allocated. Another process could then
try grow the pool, and fail to convert a surplus huge page to a normal
huge page and instead allocate a fresh huge page. I believe this is
benign, as no memory is leaked (the actual pages are still tracked
correctly) and the counters won't go out of sync.
2) Shrinking the static pool while a surplus is in effect will allow the
number of surplus huge pages to exceed the overcommit value. As long as
this condition holds, however, no more surplus huge pages will be
allowed on the system until one of the two sysctls are increased
sufficiently, or the surplus huge pages go out of use and are freed.
Successfully tested on x86_64 with the current libhugetlbfs snapshot,
modified to use the new sysctl.
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Dave Hansen <haveblue@us.ibm.com>
Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-12-18 07:20:12 +07:00
|
|
|
spin_unlock(&hugetlb_lock);
|
2007-10-16 15:26:18 +07:00
|
|
|
|
|
|
|
return page;
|
|
|
|
}
|
|
|
|
|
2007-10-16 15:26:19 +07:00
|
|
|
/*
|
|
|
|
* Increase the hugetlb pool such that it can accomodate a reservation
|
|
|
|
* of size 'delta'.
|
|
|
|
*/
|
2008-07-24 11:27:41 +07:00
|
|
|
static int gather_surplus_pages(struct hstate *h, int delta)
|
2007-10-16 15:26:19 +07:00
|
|
|
{
|
|
|
|
struct list_head surplus_list;
|
|
|
|
struct page *page, *tmp;
|
|
|
|
int ret, i;
|
|
|
|
int needed, allocated;
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
needed = (h->resv_huge_pages + delta) - h->free_huge_pages;
|
2008-03-05 05:29:38 +07:00
|
|
|
if (needed <= 0) {
|
2008-07-24 11:27:41 +07:00
|
|
|
h->resv_huge_pages += delta;
|
2007-10-16 15:26:19 +07:00
|
|
|
return 0;
|
2008-03-05 05:29:38 +07:00
|
|
|
}
|
2007-10-16 15:26:19 +07:00
|
|
|
|
|
|
|
allocated = 0;
|
|
|
|
INIT_LIST_HEAD(&surplus_list);
|
|
|
|
|
|
|
|
ret = -ENOMEM;
|
|
|
|
retry:
|
|
|
|
spin_unlock(&hugetlb_lock);
|
|
|
|
for (i = 0; i < needed; i++) {
|
2008-07-24 11:27:41 +07:00
|
|
|
page = alloc_buddy_huge_page(h, NULL, 0);
|
2007-10-16 15:26:19 +07:00
|
|
|
if (!page) {
|
|
|
|
/*
|
|
|
|
* We were not able to allocate enough pages to
|
|
|
|
* satisfy the entire reservation so we free what
|
|
|
|
* we've allocated so far.
|
|
|
|
*/
|
|
|
|
spin_lock(&hugetlb_lock);
|
|
|
|
needed = 0;
|
|
|
|
goto free;
|
|
|
|
}
|
|
|
|
|
|
|
|
list_add(&page->lru, &surplus_list);
|
|
|
|
}
|
|
|
|
allocated += needed;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* After retaking hugetlb_lock, we need to recalculate 'needed'
|
|
|
|
* because either resv_huge_pages or free_huge_pages may have changed.
|
|
|
|
*/
|
|
|
|
spin_lock(&hugetlb_lock);
|
2008-07-24 11:27:41 +07:00
|
|
|
needed = (h->resv_huge_pages + delta) -
|
|
|
|
(h->free_huge_pages + allocated);
|
2007-10-16 15:26:19 +07:00
|
|
|
if (needed > 0)
|
|
|
|
goto retry;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The surplus_list now contains _at_least_ the number of extra pages
|
|
|
|
* needed to accomodate the reservation. Add the appropriate number
|
|
|
|
* of pages to the hugetlb pool and free the extras back to the buddy
|
2008-03-05 05:29:38 +07:00
|
|
|
* allocator. Commit the entire reservation here to prevent another
|
|
|
|
* process from stealing the pages as they are added to the pool but
|
|
|
|
* before they are reserved.
|
2007-10-16 15:26:19 +07:00
|
|
|
*/
|
|
|
|
needed += allocated;
|
2008-07-24 11:27:41 +07:00
|
|
|
h->resv_huge_pages += delta;
|
2007-10-16 15:26:19 +07:00
|
|
|
ret = 0;
|
|
|
|
free:
|
2008-04-28 16:12:20 +07:00
|
|
|
/* Free the needed pages to the hugetlb pool */
|
2007-10-16 15:26:19 +07:00
|
|
|
list_for_each_entry_safe(page, tmp, &surplus_list, lru) {
|
2008-04-28 16:12:20 +07:00
|
|
|
if ((--needed) < 0)
|
|
|
|
break;
|
2007-10-16 15:26:19 +07:00
|
|
|
list_del(&page->lru);
|
2008-07-24 11:27:41 +07:00
|
|
|
enqueue_huge_page(h, page);
|
2008-04-28 16:12:20 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Free unnecessary surplus pages to the buddy allocator */
|
|
|
|
if (!list_empty(&surplus_list)) {
|
|
|
|
spin_unlock(&hugetlb_lock);
|
|
|
|
list_for_each_entry_safe(page, tmp, &surplus_list, lru) {
|
|
|
|
list_del(&page->lru);
|
2007-10-16 15:26:25 +07:00
|
|
|
/*
|
2008-03-11 01:43:50 +07:00
|
|
|
* The page has a reference count of zero already, so
|
|
|
|
* call free_huge_page directly instead of using
|
|
|
|
* put_page. This must be done with hugetlb_lock
|
2007-10-16 15:26:25 +07:00
|
|
|
* unlocked which is safe because free_huge_page takes
|
|
|
|
* hugetlb_lock before deciding how to free the page.
|
|
|
|
*/
|
2008-03-11 01:43:50 +07:00
|
|
|
free_huge_page(page);
|
2007-10-16 15:26:25 +07:00
|
|
|
}
|
2008-04-28 16:12:20 +07:00
|
|
|
spin_lock(&hugetlb_lock);
|
2007-10-16 15:26:19 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* When releasing a hugetlb pool reservation, any surplus pages that were
|
|
|
|
* allocated to satisfy the reservation must be explicitly freed if they were
|
|
|
|
* never used.
|
|
|
|
*/
|
2008-07-24 11:27:41 +07:00
|
|
|
static void return_unused_surplus_pages(struct hstate *h,
|
|
|
|
unsigned long unused_resv_pages)
|
2007-10-16 15:26:19 +07:00
|
|
|
{
|
|
|
|
static int nid = -1;
|
|
|
|
struct page *page;
|
|
|
|
unsigned long nr_pages;
|
|
|
|
|
2008-03-27 04:40:20 +07:00
|
|
|
/*
|
|
|
|
* We want to release as many surplus pages as possible, spread
|
|
|
|
* evenly across all nodes. Iterate across all nodes until we
|
|
|
|
* can no longer free unreserved surplus pages. This occurs when
|
|
|
|
* the nodes with surplus pages have no free pages.
|
|
|
|
*/
|
2009-06-17 05:32:15 +07:00
|
|
|
unsigned long remaining_iterations = nr_online_nodes;
|
2008-03-27 04:40:20 +07:00
|
|
|
|
2008-03-05 05:29:38 +07:00
|
|
|
/* Uncommit the reservation */
|
2008-07-24 11:27:41 +07:00
|
|
|
h->resv_huge_pages -= unused_resv_pages;
|
2008-03-05 05:29:38 +07:00
|
|
|
|
2008-07-24 11:27:47 +07:00
|
|
|
/* Cannot return gigantic pages currently */
|
|
|
|
if (h->order >= MAX_ORDER)
|
|
|
|
return;
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
nr_pages = min(unused_resv_pages, h->surplus_huge_pages);
|
2007-10-16 15:26:19 +07:00
|
|
|
|
2008-03-27 04:40:20 +07:00
|
|
|
while (remaining_iterations-- && nr_pages) {
|
2007-10-16 15:26:19 +07:00
|
|
|
nid = next_node(nid, node_online_map);
|
|
|
|
if (nid == MAX_NUMNODES)
|
|
|
|
nid = first_node(node_online_map);
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
if (!h->surplus_huge_pages_node[nid])
|
2007-10-16 15:26:19 +07:00
|
|
|
continue;
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
if (!list_empty(&h->hugepage_freelists[nid])) {
|
|
|
|
page = list_entry(h->hugepage_freelists[nid].next,
|
2007-10-16 15:26:19 +07:00
|
|
|
struct page, lru);
|
|
|
|
list_del(&page->lru);
|
2008-07-24 11:27:41 +07:00
|
|
|
update_and_free_page(h, page);
|
|
|
|
h->free_huge_pages--;
|
|
|
|
h->free_huge_pages_node[nid]--;
|
|
|
|
h->surplus_huge_pages--;
|
|
|
|
h->surplus_huge_pages_node[nid]--;
|
2007-10-16 15:26:19 +07:00
|
|
|
nr_pages--;
|
2009-06-17 05:32:15 +07:00
|
|
|
remaining_iterations = nr_online_nodes;
|
2007-10-16 15:26:19 +07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:30 +07:00
|
|
|
/*
|
|
|
|
* Determine if the huge page at addr within the vma has an associated
|
|
|
|
* reservation. Where it does not we will need to logically increase
|
|
|
|
* reservation and actually increase quota before an allocation can occur.
|
|
|
|
* Where any new reservation would be required the reservation change is
|
|
|
|
* prepared, but not committed. Once the page has been quota'd allocated
|
|
|
|
* an instantiated the change should be committed via vma_commit_reservation.
|
|
|
|
* No action is required on failure.
|
|
|
|
*/
|
2009-04-01 05:23:15 +07:00
|
|
|
static long vma_needs_reservation(struct hstate *h,
|
2008-07-24 11:27:41 +07:00
|
|
|
struct vm_area_struct *vma, unsigned long addr)
|
2008-07-24 11:27:30 +07:00
|
|
|
{
|
|
|
|
struct address_space *mapping = vma->vm_file->f_mapping;
|
|
|
|
struct inode *inode = mapping->host;
|
|
|
|
|
2009-05-29 04:34:40 +07:00
|
|
|
if (vma->vm_flags & VM_MAYSHARE) {
|
2008-07-24 11:27:41 +07:00
|
|
|
pgoff_t idx = vma_hugecache_offset(h, vma, addr);
|
2008-07-24 11:27:30 +07:00
|
|
|
return region_chg(&inode->i_mapping->private_list,
|
|
|
|
idx, idx + 1);
|
|
|
|
|
2008-07-24 11:27:32 +07:00
|
|
|
} else if (!is_vma_resv_set(vma, HPAGE_RESV_OWNER)) {
|
|
|
|
return 1;
|
2008-07-24 11:27:30 +07:00
|
|
|
|
2008-07-24 11:27:32 +07:00
|
|
|
} else {
|
2009-04-01 05:23:15 +07:00
|
|
|
long err;
|
2008-07-24 11:27:41 +07:00
|
|
|
pgoff_t idx = vma_hugecache_offset(h, vma, addr);
|
2008-07-24 11:27:32 +07:00
|
|
|
struct resv_map *reservations = vma_resv_map(vma);
|
|
|
|
|
|
|
|
err = region_chg(&reservations->regions, idx, idx + 1);
|
|
|
|
if (err < 0)
|
|
|
|
return err;
|
|
|
|
return 0;
|
|
|
|
}
|
2008-07-24 11:27:30 +07:00
|
|
|
}
|
2008-07-24 11:27:41 +07:00
|
|
|
static void vma_commit_reservation(struct hstate *h,
|
|
|
|
struct vm_area_struct *vma, unsigned long addr)
|
2008-07-24 11:27:30 +07:00
|
|
|
{
|
|
|
|
struct address_space *mapping = vma->vm_file->f_mapping;
|
|
|
|
struct inode *inode = mapping->host;
|
|
|
|
|
2009-05-29 04:34:40 +07:00
|
|
|
if (vma->vm_flags & VM_MAYSHARE) {
|
2008-07-24 11:27:41 +07:00
|
|
|
pgoff_t idx = vma_hugecache_offset(h, vma, addr);
|
2008-07-24 11:27:30 +07:00
|
|
|
region_add(&inode->i_mapping->private_list, idx, idx + 1);
|
2008-07-24 11:27:32 +07:00
|
|
|
|
|
|
|
} else if (is_vma_resv_set(vma, HPAGE_RESV_OWNER)) {
|
2008-07-24 11:27:41 +07:00
|
|
|
pgoff_t idx = vma_hugecache_offset(h, vma, addr);
|
2008-07-24 11:27:32 +07:00
|
|
|
struct resv_map *reservations = vma_resv_map(vma);
|
|
|
|
|
|
|
|
/* Mark this page used in the map. */
|
|
|
|
region_add(&reservations->regions, idx, idx + 1);
|
2008-07-24 11:27:30 +07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:23 +07:00
|
|
|
static struct page *alloc_huge_page(struct vm_area_struct *vma,
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
unsigned long addr, int avoid_reserve)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2008-07-24 11:27:41 +07:00
|
|
|
struct hstate *h = hstate_vma(vma);
|
2007-11-15 07:59:37 +07:00
|
|
|
struct page *page;
|
2008-07-24 11:27:23 +07:00
|
|
|
struct address_space *mapping = vma->vm_file->f_mapping;
|
|
|
|
struct inode *inode = mapping->host;
|
2009-04-01 05:23:15 +07:00
|
|
|
long chg;
|
2008-07-24 11:27:23 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Processes that did not create the mapping will have no reserves and
|
|
|
|
* will not have accounted against quota. Check that the quota can be
|
|
|
|
* made before satisfying the allocation
|
2008-07-24 11:27:30 +07:00
|
|
|
* MAP_NORESERVE mappings may also need pages and quota allocated
|
|
|
|
* if no reserve mapping overlaps.
|
2008-07-24 11:27:23 +07:00
|
|
|
*/
|
2008-07-24 11:27:41 +07:00
|
|
|
chg = vma_needs_reservation(h, vma, addr);
|
2008-07-24 11:27:30 +07:00
|
|
|
if (chg < 0)
|
|
|
|
return ERR_PTR(chg);
|
|
|
|
if (chg)
|
2008-07-24 11:27:23 +07:00
|
|
|
if (hugetlb_get_quota(inode->i_mapping, chg))
|
|
|
|
return ERR_PTR(-ENOSPC);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
spin_lock(&hugetlb_lock);
|
2008-07-24 11:27:41 +07:00
|
|
|
page = dequeue_huge_page_vma(h, vma, addr, avoid_reserve);
|
2005-04-17 05:20:36 +07:00
|
|
|
spin_unlock(&hugetlb_lock);
|
[PATCH] hugepage: Strict page reservation for hugepage inodes
These days, hugepages are demand-allocated at first fault time. There's a
somewhat dubious (and racy) heuristic when making a new mmap() to check if
there are enough available hugepages to fully satisfy that mapping.
A particularly obvious case where the heuristic breaks down is where a
process maps its hugepages not as a single chunk, but as a bunch of
individually mmap()ed (or shmat()ed) blocks without touching and
instantiating the pages in between allocations. In this case the size of
each block is compared against the total number of available hugepages.
It's thus easy for the process to become overcommitted, because each block
mapping will succeed, although the total number of hugepages required by
all blocks exceeds the number available. In particular, this defeats such
a program which will detect a mapping failure and adjust its hugepage usage
downward accordingly.
The patch below addresses this problem, by strictly reserving a number of
physical hugepages for hugepage inodes which have been mapped, but not
instatiated. MAP_SHARED mappings are thus "safe" - they will fail on
mmap(), not later with an OOM SIGKILL. MAP_PRIVATE mappings can still
trigger an OOM. (Actually SHARED mappings can technically still OOM, but
only if the sysadmin explicitly reduces the hugepage pool between mapping
and instantiation)
This patch appears to address the problem at hand - it allows DB2 to start
correctly, for instance, which previously suffered the failure described
above.
This patch causes no regressions on the libhugetblfs testsuite, and makes a
test (designed to catch this problem) pass which previously failed (ppc64,
POWER5).
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-03-22 15:08:55 +07:00
|
|
|
|
2008-01-14 15:55:19 +07:00
|
|
|
if (!page) {
|
2008-07-24 11:27:41 +07:00
|
|
|
page = alloc_buddy_huge_page(h, vma, addr);
|
2008-01-14 15:55:19 +07:00
|
|
|
if (!page) {
|
2008-07-24 11:27:23 +07:00
|
|
|
hugetlb_put_quota(inode->i_mapping, chg);
|
2008-01-14 15:55:19 +07:00
|
|
|
return ERR_PTR(-VM_FAULT_OOM);
|
|
|
|
}
|
|
|
|
}
|
2007-11-15 07:59:37 +07:00
|
|
|
|
2008-07-24 11:27:23 +07:00
|
|
|
set_page_refcounted(page);
|
|
|
|
set_page_private(page, (unsigned long) mapping);
|
2007-11-15 07:59:42 +07:00
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
vma_commit_reservation(h, vma, addr);
|
2008-07-24 11:27:30 +07:00
|
|
|
|
2007-11-15 07:59:42 +07:00
|
|
|
return page;
|
[PATCH] hugepage: Strict page reservation for hugepage inodes
These days, hugepages are demand-allocated at first fault time. There's a
somewhat dubious (and racy) heuristic when making a new mmap() to check if
there are enough available hugepages to fully satisfy that mapping.
A particularly obvious case where the heuristic breaks down is where a
process maps its hugepages not as a single chunk, but as a bunch of
individually mmap()ed (or shmat()ed) blocks without touching and
instantiating the pages in between allocations. In this case the size of
each block is compared against the total number of available hugepages.
It's thus easy for the process to become overcommitted, because each block
mapping will succeed, although the total number of hugepages required by
all blocks exceeds the number available. In particular, this defeats such
a program which will detect a mapping failure and adjust its hugepage usage
downward accordingly.
The patch below addresses this problem, by strictly reserving a number of
physical hugepages for hugepage inodes which have been mapped, but not
instatiated. MAP_SHARED mappings are thus "safe" - they will fail on
mmap(), not later with an OOM SIGKILL. MAP_PRIVATE mappings can still
trigger an OOM. (Actually SHARED mappings can technically still OOM, but
only if the sysadmin explicitly reduces the hugepage pool between mapping
and instantiation)
This patch appears to address the problem at hand - it allows DB2 to start
correctly, for instance, which previously suffered the failure described
above.
This patch causes no regressions on the libhugetblfs testsuite, and makes a
test (designed to catch this problem) pass which previously failed (ppc64,
POWER5).
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-03-22 15:08:55 +07:00
|
|
|
}
|
|
|
|
|
2009-01-07 05:40:33 +07:00
|
|
|
int __weak alloc_bootmem_huge_page(struct hstate *h)
|
2008-07-24 11:27:47 +07:00
|
|
|
{
|
|
|
|
struct huge_bootmem_page *m;
|
|
|
|
int nr_nodes = nodes_weight(node_online_map);
|
|
|
|
|
|
|
|
while (nr_nodes) {
|
|
|
|
void *addr;
|
|
|
|
|
|
|
|
addr = __alloc_bootmem_node_nopanic(
|
|
|
|
NODE_DATA(h->hugetlb_next_nid),
|
|
|
|
huge_page_size(h), huge_page_size(h), 0);
|
|
|
|
|
|
|
|
if (addr) {
|
|
|
|
/*
|
|
|
|
* Use the beginning of the huge page to store the
|
|
|
|
* huge_bootmem_page struct (until gather_bootmem
|
|
|
|
* puts them into the mem_map).
|
|
|
|
*/
|
|
|
|
m = addr;
|
2009-01-07 05:40:33 +07:00
|
|
|
goto found;
|
2008-07-24 11:27:47 +07:00
|
|
|
}
|
|
|
|
hstate_next_node(h);
|
|
|
|
nr_nodes--;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
found:
|
|
|
|
BUG_ON((unsigned long)virt_to_phys(m) & (huge_page_size(h) - 1));
|
|
|
|
/* Put them into a private list first because mem_map is not up yet */
|
|
|
|
list_add(&m->list, &huge_boot_pages);
|
|
|
|
m->hstate = h;
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2008-11-07 03:53:27 +07:00
|
|
|
static void prep_compound_huge_page(struct page *page, int order)
|
|
|
|
{
|
|
|
|
if (unlikely(order > (MAX_ORDER - 1)))
|
|
|
|
prep_compound_gigantic_page(page, order);
|
|
|
|
else
|
|
|
|
prep_compound_page(page, order);
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:47 +07:00
|
|
|
/* Put bootmem huge pages into the standard lists after mem_map is up */
|
|
|
|
static void __init gather_bootmem_prealloc(void)
|
|
|
|
{
|
|
|
|
struct huge_bootmem_page *m;
|
|
|
|
|
|
|
|
list_for_each_entry(m, &huge_boot_pages, list) {
|
|
|
|
struct page *page = virt_to_page(m);
|
|
|
|
struct hstate *h = m->hstate;
|
|
|
|
__ClearPageReserved(page);
|
|
|
|
WARN_ON(page_count(page) != 1);
|
2008-11-07 03:53:27 +07:00
|
|
|
prep_compound_huge_page(page, h->order);
|
2008-07-24 11:27:47 +07:00
|
|
|
prep_new_huge_page(h, page, page_to_nid(page));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:48 +07:00
|
|
|
static void __init hugetlb_hstate_alloc_pages(struct hstate *h)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
unsigned long i;
|
2008-07-24 11:27:41 +07:00
|
|
|
|
2008-07-24 11:27:42 +07:00
|
|
|
for (i = 0; i < h->max_huge_pages; ++i) {
|
2008-07-24 11:27:47 +07:00
|
|
|
if (h->order >= MAX_ORDER) {
|
|
|
|
if (!alloc_bootmem_huge_page(h))
|
|
|
|
break;
|
|
|
|
} else if (!alloc_fresh_huge_page(h))
|
2005-04-17 05:20:36 +07:00
|
|
|
break;
|
|
|
|
}
|
2008-07-24 11:27:48 +07:00
|
|
|
h->max_huge_pages = i;
|
2008-07-24 11:27:42 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
static void __init hugetlb_init_hstates(void)
|
|
|
|
{
|
|
|
|
struct hstate *h;
|
|
|
|
|
|
|
|
for_each_hstate(h) {
|
2008-07-24 11:27:48 +07:00
|
|
|
/* oversize hugepages were init'ed in early boot */
|
|
|
|
if (h->order < MAX_ORDER)
|
|
|
|
hugetlb_hstate_alloc_pages(h);
|
2008-07-24 11:27:42 +07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:49 +07:00
|
|
|
static char * __init memfmt(char *buf, unsigned long n)
|
|
|
|
{
|
|
|
|
if (n >= (1UL << 30))
|
|
|
|
sprintf(buf, "%lu GB", n >> 30);
|
|
|
|
else if (n >= (1UL << 20))
|
|
|
|
sprintf(buf, "%lu MB", n >> 20);
|
|
|
|
else
|
|
|
|
sprintf(buf, "%lu KB", n >> 10);
|
|
|
|
return buf;
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:42 +07:00
|
|
|
static void __init report_hugepages(void)
|
|
|
|
{
|
|
|
|
struct hstate *h;
|
|
|
|
|
|
|
|
for_each_hstate(h) {
|
2008-07-24 11:27:49 +07:00
|
|
|
char buf[32];
|
|
|
|
printk(KERN_INFO "HugeTLB registered %s page size, "
|
|
|
|
"pre-allocated %ld pages\n",
|
|
|
|
memfmt(buf, huge_page_size(h)),
|
|
|
|
h->free_huge_pages);
|
2008-07-24 11:27:42 +07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
#ifdef CONFIG_HIGHMEM
|
2008-07-24 11:27:41 +07:00
|
|
|
static void try_to_free_low(struct hstate *h, unsigned long count)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2006-09-26 13:31:55 +07:00
|
|
|
int i;
|
|
|
|
|
2008-07-24 11:27:47 +07:00
|
|
|
if (h->order >= MAX_ORDER)
|
|
|
|
return;
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
for (i = 0; i < MAX_NUMNODES; ++i) {
|
|
|
|
struct page *page, *next;
|
2008-07-24 11:27:41 +07:00
|
|
|
struct list_head *freel = &h->hugepage_freelists[i];
|
|
|
|
list_for_each_entry_safe(page, next, freel, lru) {
|
|
|
|
if (count >= h->nr_huge_pages)
|
2007-10-16 15:26:23 +07:00
|
|
|
return;
|
2005-04-17 05:20:36 +07:00
|
|
|
if (PageHighMem(page))
|
|
|
|
continue;
|
|
|
|
list_del(&page->lru);
|
2008-07-24 11:27:42 +07:00
|
|
|
update_and_free_page(h, page);
|
2008-07-24 11:27:41 +07:00
|
|
|
h->free_huge_pages--;
|
|
|
|
h->free_huge_pages_node[page_to_nid(page)]--;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
#else
|
2008-07-24 11:27:41 +07:00
|
|
|
static inline void try_to_free_low(struct hstate *h, unsigned long count)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
mm: introduce PageHuge() for testing huge/gigantic pages
A series of patches to enhance the /proc/pagemap interface and to add a
userspace executable which can be used to present the pagemap data.
Export 10 more flags to end users (and more for kernel developers):
11. KPF_MMAP (pseudo flag) memory mapped page
12. KPF_ANON (pseudo flag) memory mapped page (anonymous)
13. KPF_SWAPCACHE page is in swap cache
14. KPF_SWAPBACKED page is swap/RAM backed
15. KPF_COMPOUND_HEAD (*)
16. KPF_COMPOUND_TAIL (*)
17. KPF_HUGE hugeTLB pages
18. KPF_UNEVICTABLE page is in the unevictable LRU list
19. KPF_HWPOISON hardware detected corruption
20. KPF_NOPAGE (pseudo flag) no page frame at the address
(*) For compound pages, exporting _both_ head/tail info enables
users to tell where a compound page starts/ends, and its order.
a simple demo of the page-types tool
# ./page-types -h
page-types [options]
-r|--raw Raw mode, for kernel developers
-a|--addr addr-spec Walk a range of pages
-b|--bits bits-spec Walk pages with specified bits
-l|--list Show page details in ranges
-L|--list-each Show page details one by one
-N|--no-summary Don't show summay info
-h|--help Show this usage message
addr-spec:
N one page at offset N (unit: pages)
N+M pages range from N to N+M-1
N,M pages range from N to M-1
N, pages range from N to end
,M pages range from 0 to M
bits-spec:
bit1,bit2 (flags & (bit1|bit2)) != 0
bit1,bit2=bit1 (flags & (bit1|bit2)) == bit1
bit1,~bit2 (flags & (bit1|bit2)) == bit1
=bit1,bit2 flags == (bit1|bit2)
bit-names:
locked error referenced uptodate
dirty lru active slab
writeback reclaim buddy mmap
anonymous swapcache swapbacked compound_head
compound_tail huge unevictable hwpoison
nopage reserved(r) mlocked(r) mappedtodisk(r)
private(r) private_2(r) owner_private(r) arch(r)
uncached(r) readahead(o) slob_free(o) slub_frozen(o)
slub_debug(o)
(r) raw mode bits (o) overloaded bits
# ./page-types
flags page-count MB symbolic-flags long-symbolic-flags
0x0000000000000000 487369 1903 _________________________________
0x0000000000000014 5 0 __R_D____________________________ referenced,dirty
0x0000000000000020 1 0 _____l___________________________ lru
0x0000000000000024 34 0 __R__l___________________________ referenced,lru
0x0000000000000028 3838 14 ___U_l___________________________ uptodate,lru
0x0001000000000028 48 0 ___U_l_______________________I___ uptodate,lru,readahead
0x000000000000002c 6478 25 __RU_l___________________________ referenced,uptodate,lru
0x000100000000002c 47 0 __RU_l_______________________I___ referenced,uptodate,lru,readahead
0x0000000000000040 8344 32 ______A__________________________ active
0x0000000000000060 1 0 _____lA__________________________ lru,active
0x0000000000000068 348 1 ___U_lA__________________________ uptodate,lru,active
0x0001000000000068 12 0 ___U_lA______________________I___ uptodate,lru,active,readahead
0x000000000000006c 988 3 __RU_lA__________________________ referenced,uptodate,lru,active
0x000100000000006c 48 0 __RU_lA______________________I___ referenced,uptodate,lru,active,readahead
0x0000000000004078 1 0 ___UDlA_______b__________________ uptodate,dirty,lru,active,swapbacked
0x000000000000407c 34 0 __RUDlA_______b__________________ referenced,uptodate,dirty,lru,active,swapbacked
0x0000000000000400 503 1 __________B______________________ buddy
0x0000000000000804 1 0 __R________M_____________________ referenced,mmap
0x0000000000000828 1029 4 ___U_l_____M_____________________ uptodate,lru,mmap
0x0001000000000828 43 0 ___U_l_____M_________________I___ uptodate,lru,mmap,readahead
0x000000000000082c 382 1 __RU_l_____M_____________________ referenced,uptodate,lru,mmap
0x000100000000082c 12 0 __RU_l_____M_________________I___ referenced,uptodate,lru,mmap,readahead
0x0000000000000868 192 0 ___U_lA____M_____________________ uptodate,lru,active,mmap
0x0001000000000868 12 0 ___U_lA____M_________________I___ uptodate,lru,active,mmap,readahead
0x000000000000086c 800 3 __RU_lA____M_____________________ referenced,uptodate,lru,active,mmap
0x000100000000086c 31 0 __RU_lA____M_________________I___ referenced,uptodate,lru,active,mmap,readahead
0x0000000000004878 2 0 ___UDlA____M__b__________________ uptodate,dirty,lru,active,mmap,swapbacked
0x0000000000001000 492 1 ____________a____________________ anonymous
0x0000000000005808 4 0 ___U_______Ma_b__________________ uptodate,mmap,anonymous,swapbacked
0x0000000000005868 2839 11 ___U_lA____Ma_b__________________ uptodate,lru,active,mmap,anonymous,swapbacked
0x000000000000586c 30 0 __RU_lA____Ma_b__________________ referenced,uptodate,lru,active,mmap,anonymous,swapbacked
total 513968 2007
# ./page-types -r
flags page-count MB symbolic-flags long-symbolic-flags
0x0000000000000000 468002 1828 _________________________________
0x0000000100000000 19102 74 _____________________r___________ reserved
0x0000000000008000 41 0 _______________H_________________ compound_head
0x0000000000010000 188 0 ________________T________________ compound_tail
0x0000000000008014 1 0 __R_D__________H_________________ referenced,dirty,compound_head
0x0000000000010014 4 0 __R_D___________T________________ referenced,dirty,compound_tail
0x0000000000000020 1 0 _____l___________________________ lru
0x0000000800000024 34 0 __R__l__________________P________ referenced,lru,private
0x0000000000000028 3794 14 ___U_l___________________________ uptodate,lru
0x0001000000000028 46 0 ___U_l_______________________I___ uptodate,lru,readahead
0x0000000400000028 44 0 ___U_l_________________d_________ uptodate,lru,mappedtodisk
0x0001000400000028 2 0 ___U_l_________________d_____I___ uptodate,lru,mappedtodisk,readahead
0x000000000000002c 6434 25 __RU_l___________________________ referenced,uptodate,lru
0x000100000000002c 47 0 __RU_l_______________________I___ referenced,uptodate,lru,readahead
0x000000040000002c 14 0 __RU_l_________________d_________ referenced,uptodate,lru,mappedtodisk
0x000000080000002c 30 0 __RU_l__________________P________ referenced,uptodate,lru,private
0x0000000800000040 8124 31 ______A_________________P________ active,private
0x0000000000000040 219 0 ______A__________________________ active
0x0000000800000060 1 0 _____lA_________________P________ lru,active,private
0x0000000000000068 322 1 ___U_lA__________________________ uptodate,lru,active
0x0001000000000068 12 0 ___U_lA______________________I___ uptodate,lru,active,readahead
0x0000000400000068 13 0 ___U_lA________________d_________ uptodate,lru,active,mappedtodisk
0x0000000800000068 12 0 ___U_lA_________________P________ uptodate,lru,active,private
0x000000000000006c 977 3 __RU_lA__________________________ referenced,uptodate,lru,active
0x000100000000006c 48 0 __RU_lA______________________I___ referenced,uptodate,lru,active,readahead
0x000000040000006c 5 0 __RU_lA________________d_________ referenced,uptodate,lru,active,mappedtodisk
0x000000080000006c 3 0 __RU_lA_________________P________ referenced,uptodate,lru,active,private
0x0000000c0000006c 3 0 __RU_lA________________dP________ referenced,uptodate,lru,active,mappedtodisk,private
0x0000000c00000068 1 0 ___U_lA________________dP________ uptodate,lru,active,mappedtodisk,private
0x0000000000004078 1 0 ___UDlA_______b__________________ uptodate,dirty,lru,active,swapbacked
0x000000000000407c 34 0 __RUDlA_______b__________________ referenced,uptodate,dirty,lru,active,swapbacked
0x0000000000000400 538 2 __________B______________________ buddy
0x0000000000000804 1 0 __R________M_____________________ referenced,mmap
0x0000000000000828 1029 4 ___U_l_____M_____________________ uptodate,lru,mmap
0x0001000000000828 43 0 ___U_l_____M_________________I___ uptodate,lru,mmap,readahead
0x000000000000082c 382 1 __RU_l_____M_____________________ referenced,uptodate,lru,mmap
0x000100000000082c 12 0 __RU_l_____M_________________I___ referenced,uptodate,lru,mmap,readahead
0x0000000000000868 192 0 ___U_lA____M_____________________ uptodate,lru,active,mmap
0x0001000000000868 12 0 ___U_lA____M_________________I___ uptodate,lru,active,mmap,readahead
0x000000000000086c 800 3 __RU_lA____M_____________________ referenced,uptodate,lru,active,mmap
0x000100000000086c 31 0 __RU_lA____M_________________I___ referenced,uptodate,lru,active,mmap,readahead
0x0000000000004878 2 0 ___UDlA____M__b__________________ uptodate,dirty,lru,active,mmap,swapbacked
0x0000000000001000 492 1 ____________a____________________ anonymous
0x0000000000005008 2 0 ___U________a_b__________________ uptodate,anonymous,swapbacked
0x0000000000005808 4 0 ___U_______Ma_b__________________ uptodate,mmap,anonymous,swapbacked
0x000000000000580c 1 0 __RU_______Ma_b__________________ referenced,uptodate,mmap,anonymous,swapbacked
0x0000000000005868 2839 11 ___U_lA____Ma_b__________________ uptodate,lru,active,mmap,anonymous,swapbacked
0x000000000000586c 29 0 __RU_lA____Ma_b__________________ referenced,uptodate,lru,active,mmap,anonymous,swapbacked
total 513968 2007
# ./page-types --raw --list --no-summary --bits reserved
offset count flags
0 15 _____________________r___________
31 4 _____________________r___________
159 97 _____________________r___________
4096 2067 _____________________r___________
6752 2390 _____________________r___________
9355 3 _____________________r___________
9728 14526 _____________________r___________
This patch:
Introduce PageHuge(), which identifies huge/gigantic pages by their
dedicated compound destructor functions.
Also move prep_compound_gigantic_page() to hugetlb.c and make
__free_pages_ok() non-static.
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: Matt Mackall <mpm@selenic.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2009-06-17 05:32:22 +07:00
|
|
|
/*
|
|
|
|
* Increment or decrement surplus_huge_pages. Keep node-specific counters
|
|
|
|
* balanced by operating on them in a round-robin fashion.
|
|
|
|
* Returns 1 if an adjustment was made.
|
|
|
|
*/
|
|
|
|
static int adjust_pool_surplus(struct hstate *h, int delta)
|
|
|
|
{
|
|
|
|
static int prev_nid;
|
|
|
|
int nid = prev_nid;
|
|
|
|
int ret = 0;
|
|
|
|
|
|
|
|
VM_BUG_ON(delta != -1 && delta != 1);
|
|
|
|
do {
|
|
|
|
nid = next_node(nid, node_online_map);
|
|
|
|
if (nid == MAX_NUMNODES)
|
|
|
|
nid = first_node(node_online_map);
|
|
|
|
|
|
|
|
/* To shrink on this node, there must be a surplus page */
|
|
|
|
if (delta < 0 && !h->surplus_huge_pages_node[nid])
|
|
|
|
continue;
|
|
|
|
/* Surplus cannot exceed the total number of pages */
|
|
|
|
if (delta > 0 && h->surplus_huge_pages_node[nid] >=
|
|
|
|
h->nr_huge_pages_node[nid])
|
|
|
|
continue;
|
|
|
|
|
|
|
|
h->surplus_huge_pages += delta;
|
|
|
|
h->surplus_huge_pages_node[nid] += delta;
|
|
|
|
ret = 1;
|
|
|
|
break;
|
|
|
|
} while (nid != prev_nid);
|
|
|
|
|
|
|
|
prev_nid = nid;
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
#define persistent_huge_pages(h) (h->nr_huge_pages - h->surplus_huge_pages)
|
2008-07-24 11:27:42 +07:00
|
|
|
static unsigned long set_max_huge_pages(struct hstate *h, unsigned long count)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2007-10-16 15:26:18 +07:00
|
|
|
unsigned long min_count, ret;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2008-07-24 11:27:47 +07:00
|
|
|
if (h->order >= MAX_ORDER)
|
|
|
|
return h->max_huge_pages;
|
|
|
|
|
2007-10-16 15:26:18 +07:00
|
|
|
/*
|
|
|
|
* Increase the pool size
|
|
|
|
* First take pages out of surplus state. Then make up the
|
|
|
|
* remaining difference by allocating fresh huge pages.
|
hugetlb: introduce nr_overcommit_hugepages sysctl
hugetlb: introduce nr_overcommit_hugepages sysctl
While examining the code to support /proc/sys/vm/hugetlb_dynamic_pool, I
became convinced that having a boolean sysctl was insufficient:
1) To support per-node control of hugepages, I have previously submitted
patches to add a sysfs attribute related to nr_hugepages. However, with
a boolean global value and per-mount quota enforcement constraining the
dynamic pool, adding corresponding control of the dynamic pool on a
per-node basis seems inconsistent to me.
2) Administration of the hugetlb dynamic pool with multiple hugetlbfs
mount points is, arguably, more arduous than it needs to be. Each quota
would need to be set separately, and the sum would need to be monitored.
To ease the administration, and to help make the way for per-node
control of the static & dynamic hugepage pool, I added a separate
sysctl, nr_overcommit_hugepages. This value serves as a high watermark
for the overall hugepage pool, while nr_hugepages serves as a low
watermark. The boolean sysctl can then be removed, as the condition
nr_overcommit_hugepages > 0
indicates the same administrative setting as
hugetlb_dynamic_pool == 1
Quotas still serve as local enforcement of the size of the pool on a
per-mount basis.
A few caveats:
1) There is a race whereby the global surplus huge page counter is
incremented before a hugepage has allocated. Another process could then
try grow the pool, and fail to convert a surplus huge page to a normal
huge page and instead allocate a fresh huge page. I believe this is
benign, as no memory is leaked (the actual pages are still tracked
correctly) and the counters won't go out of sync.
2) Shrinking the static pool while a surplus is in effect will allow the
number of surplus huge pages to exceed the overcommit value. As long as
this condition holds, however, no more surplus huge pages will be
allowed on the system until one of the two sysctls are increased
sufficiently, or the surplus huge pages go out of use and are freed.
Successfully tested on x86_64 with the current libhugetlbfs snapshot,
modified to use the new sysctl.
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Dave Hansen <haveblue@us.ibm.com>
Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-12-18 07:20:12 +07:00
|
|
|
*
|
|
|
|
* We might race with alloc_buddy_huge_page() here and be unable
|
|
|
|
* to convert a surplus huge page to a normal huge page. That is
|
|
|
|
* not critical, though, it just means the overall size of the
|
|
|
|
* pool might be one hugepage larger than it needs to be, but
|
|
|
|
* within all the constraints specified by the sysctls.
|
2007-10-16 15:26:18 +07:00
|
|
|
*/
|
2005-04-17 05:20:36 +07:00
|
|
|
spin_lock(&hugetlb_lock);
|
2008-07-24 11:27:41 +07:00
|
|
|
while (h->surplus_huge_pages && count > persistent_huge_pages(h)) {
|
|
|
|
if (!adjust_pool_surplus(h, -1))
|
2007-10-16 15:26:18 +07:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
while (count > persistent_huge_pages(h)) {
|
2007-10-16 15:26:18 +07:00
|
|
|
/*
|
|
|
|
* If this allocation races such that we no longer need the
|
|
|
|
* page, free_huge_page will handle it by freeing the page
|
|
|
|
* and reducing the surplus.
|
|
|
|
*/
|
|
|
|
spin_unlock(&hugetlb_lock);
|
2008-07-24 11:27:41 +07:00
|
|
|
ret = alloc_fresh_huge_page(h);
|
2007-10-16 15:26:18 +07:00
|
|
|
spin_lock(&hugetlb_lock);
|
|
|
|
if (!ret)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Decrease the pool size
|
|
|
|
* First return free pages to the buddy allocator (being careful
|
|
|
|
* to keep enough around to satisfy reservations). Then place
|
|
|
|
* pages into surplus state as needed so the pool will shrink
|
|
|
|
* to the desired size as pages become free.
|
hugetlb: introduce nr_overcommit_hugepages sysctl
hugetlb: introduce nr_overcommit_hugepages sysctl
While examining the code to support /proc/sys/vm/hugetlb_dynamic_pool, I
became convinced that having a boolean sysctl was insufficient:
1) To support per-node control of hugepages, I have previously submitted
patches to add a sysfs attribute related to nr_hugepages. However, with
a boolean global value and per-mount quota enforcement constraining the
dynamic pool, adding corresponding control of the dynamic pool on a
per-node basis seems inconsistent to me.
2) Administration of the hugetlb dynamic pool with multiple hugetlbfs
mount points is, arguably, more arduous than it needs to be. Each quota
would need to be set separately, and the sum would need to be monitored.
To ease the administration, and to help make the way for per-node
control of the static & dynamic hugepage pool, I added a separate
sysctl, nr_overcommit_hugepages. This value serves as a high watermark
for the overall hugepage pool, while nr_hugepages serves as a low
watermark. The boolean sysctl can then be removed, as the condition
nr_overcommit_hugepages > 0
indicates the same administrative setting as
hugetlb_dynamic_pool == 1
Quotas still serve as local enforcement of the size of the pool on a
per-mount basis.
A few caveats:
1) There is a race whereby the global surplus huge page counter is
incremented before a hugepage has allocated. Another process could then
try grow the pool, and fail to convert a surplus huge page to a normal
huge page and instead allocate a fresh huge page. I believe this is
benign, as no memory is leaked (the actual pages are still tracked
correctly) and the counters won't go out of sync.
2) Shrinking the static pool while a surplus is in effect will allow the
number of surplus huge pages to exceed the overcommit value. As long as
this condition holds, however, no more surplus huge pages will be
allowed on the system until one of the two sysctls are increased
sufficiently, or the surplus huge pages go out of use and are freed.
Successfully tested on x86_64 with the current libhugetlbfs snapshot,
modified to use the new sysctl.
Signed-off-by: Nishanth Aravamudan <nacc@us.ibm.com>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Dave Hansen <haveblue@us.ibm.com>
Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-12-18 07:20:12 +07:00
|
|
|
*
|
|
|
|
* By placing pages into the surplus state independent of the
|
|
|
|
* overcommit value, we are allowing the surplus pool size to
|
|
|
|
* exceed overcommit. There are few sane options here. Since
|
|
|
|
* alloc_buddy_huge_page() is checking the global counter,
|
|
|
|
* though, we'll note that we're not allowed to exceed surplus
|
|
|
|
* and won't grow the pool anywhere else. Not until one of the
|
|
|
|
* sysctls are changed, or the surplus pages go out of use.
|
2007-10-16 15:26:18 +07:00
|
|
|
*/
|
2008-07-24 11:27:41 +07:00
|
|
|
min_count = h->resv_huge_pages + h->nr_huge_pages - h->free_huge_pages;
|
2007-10-16 15:26:23 +07:00
|
|
|
min_count = max(count, min_count);
|
2008-07-24 11:27:41 +07:00
|
|
|
try_to_free_low(h, min_count);
|
|
|
|
while (min_count < persistent_huge_pages(h)) {
|
|
|
|
struct page *page = dequeue_huge_page(h);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (!page)
|
|
|
|
break;
|
2008-07-24 11:27:41 +07:00
|
|
|
update_and_free_page(h, page);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
2008-07-24 11:27:41 +07:00
|
|
|
while (count < persistent_huge_pages(h)) {
|
|
|
|
if (!adjust_pool_surplus(h, 1))
|
2007-10-16 15:26:18 +07:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
out:
|
2008-07-24 11:27:41 +07:00
|
|
|
ret = persistent_huge_pages(h);
|
2005-04-17 05:20:36 +07:00
|
|
|
spin_unlock(&hugetlb_lock);
|
2007-10-16 15:26:18 +07:00
|
|
|
return ret;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:44 +07:00
|
|
|
#define HSTATE_ATTR_RO(_name) \
|
|
|
|
static struct kobj_attribute _name##_attr = __ATTR_RO(_name)
|
|
|
|
|
|
|
|
#define HSTATE_ATTR(_name) \
|
|
|
|
static struct kobj_attribute _name##_attr = \
|
|
|
|
__ATTR(_name, 0644, _name##_show, _name##_store)
|
|
|
|
|
|
|
|
static struct kobject *hugepages_kobj;
|
|
|
|
static struct kobject *hstate_kobjs[HUGE_MAX_HSTATE];
|
|
|
|
|
|
|
|
static struct hstate *kobj_to_hstate(struct kobject *kobj)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
for (i = 0; i < HUGE_MAX_HSTATE; i++)
|
|
|
|
if (hstate_kobjs[i] == kobj)
|
|
|
|
return &hstates[i];
|
|
|
|
BUG();
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
static ssize_t nr_hugepages_show(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr, char *buf)
|
|
|
|
{
|
|
|
|
struct hstate *h = kobj_to_hstate(kobj);
|
|
|
|
return sprintf(buf, "%lu\n", h->nr_huge_pages);
|
|
|
|
}
|
|
|
|
static ssize_t nr_hugepages_store(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr, const char *buf, size_t count)
|
|
|
|
{
|
|
|
|
int err;
|
|
|
|
unsigned long input;
|
|
|
|
struct hstate *h = kobj_to_hstate(kobj);
|
|
|
|
|
|
|
|
err = strict_strtoul(buf, 10, &input);
|
|
|
|
if (err)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
h->max_huge_pages = set_max_huge_pages(h, input);
|
|
|
|
|
|
|
|
return count;
|
|
|
|
}
|
|
|
|
HSTATE_ATTR(nr_hugepages);
|
|
|
|
|
|
|
|
static ssize_t nr_overcommit_hugepages_show(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr, char *buf)
|
|
|
|
{
|
|
|
|
struct hstate *h = kobj_to_hstate(kobj);
|
|
|
|
return sprintf(buf, "%lu\n", h->nr_overcommit_huge_pages);
|
|
|
|
}
|
|
|
|
static ssize_t nr_overcommit_hugepages_store(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr, const char *buf, size_t count)
|
|
|
|
{
|
|
|
|
int err;
|
|
|
|
unsigned long input;
|
|
|
|
struct hstate *h = kobj_to_hstate(kobj);
|
|
|
|
|
|
|
|
err = strict_strtoul(buf, 10, &input);
|
|
|
|
if (err)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
spin_lock(&hugetlb_lock);
|
|
|
|
h->nr_overcommit_huge_pages = input;
|
|
|
|
spin_unlock(&hugetlb_lock);
|
|
|
|
|
|
|
|
return count;
|
|
|
|
}
|
|
|
|
HSTATE_ATTR(nr_overcommit_hugepages);
|
|
|
|
|
|
|
|
static ssize_t free_hugepages_show(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr, char *buf)
|
|
|
|
{
|
|
|
|
struct hstate *h = kobj_to_hstate(kobj);
|
|
|
|
return sprintf(buf, "%lu\n", h->free_huge_pages);
|
|
|
|
}
|
|
|
|
HSTATE_ATTR_RO(free_hugepages);
|
|
|
|
|
|
|
|
static ssize_t resv_hugepages_show(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr, char *buf)
|
|
|
|
{
|
|
|
|
struct hstate *h = kobj_to_hstate(kobj);
|
|
|
|
return sprintf(buf, "%lu\n", h->resv_huge_pages);
|
|
|
|
}
|
|
|
|
HSTATE_ATTR_RO(resv_hugepages);
|
|
|
|
|
|
|
|
static ssize_t surplus_hugepages_show(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr, char *buf)
|
|
|
|
{
|
|
|
|
struct hstate *h = kobj_to_hstate(kobj);
|
|
|
|
return sprintf(buf, "%lu\n", h->surplus_huge_pages);
|
|
|
|
}
|
|
|
|
HSTATE_ATTR_RO(surplus_hugepages);
|
|
|
|
|
|
|
|
static struct attribute *hstate_attrs[] = {
|
|
|
|
&nr_hugepages_attr.attr,
|
|
|
|
&nr_overcommit_hugepages_attr.attr,
|
|
|
|
&free_hugepages_attr.attr,
|
|
|
|
&resv_hugepages_attr.attr,
|
|
|
|
&surplus_hugepages_attr.attr,
|
|
|
|
NULL,
|
|
|
|
};
|
|
|
|
|
|
|
|
static struct attribute_group hstate_attr_group = {
|
|
|
|
.attrs = hstate_attrs,
|
|
|
|
};
|
|
|
|
|
|
|
|
static int __init hugetlb_sysfs_add_hstate(struct hstate *h)
|
|
|
|
{
|
|
|
|
int retval;
|
|
|
|
|
|
|
|
hstate_kobjs[h - hstates] = kobject_create_and_add(h->name,
|
|
|
|
hugepages_kobj);
|
|
|
|
if (!hstate_kobjs[h - hstates])
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
retval = sysfs_create_group(hstate_kobjs[h - hstates],
|
|
|
|
&hstate_attr_group);
|
|
|
|
if (retval)
|
|
|
|
kobject_put(hstate_kobjs[h - hstates]);
|
|
|
|
|
|
|
|
return retval;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void __init hugetlb_sysfs_init(void)
|
|
|
|
{
|
|
|
|
struct hstate *h;
|
|
|
|
int err;
|
|
|
|
|
|
|
|
hugepages_kobj = kobject_create_and_add("hugepages", mm_kobj);
|
|
|
|
if (!hugepages_kobj)
|
|
|
|
return;
|
|
|
|
|
|
|
|
for_each_hstate(h) {
|
|
|
|
err = hugetlb_sysfs_add_hstate(h);
|
|
|
|
if (err)
|
|
|
|
printk(KERN_ERR "Hugetlb: Unable to add hstate %s",
|
|
|
|
h->name);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void __exit hugetlb_exit(void)
|
|
|
|
{
|
|
|
|
struct hstate *h;
|
|
|
|
|
|
|
|
for_each_hstate(h) {
|
|
|
|
kobject_put(hstate_kobjs[h - hstates]);
|
|
|
|
}
|
|
|
|
|
|
|
|
kobject_put(hugepages_kobj);
|
|
|
|
}
|
|
|
|
module_exit(hugetlb_exit);
|
|
|
|
|
|
|
|
static int __init hugetlb_init(void)
|
|
|
|
{
|
2008-07-31 14:07:30 +07:00
|
|
|
/* Some platform decide whether they support huge pages at boot
|
|
|
|
* time. On these, such as powerpc, HPAGE_SHIFT is set to 0 when
|
|
|
|
* there is no such support
|
|
|
|
*/
|
|
|
|
if (HPAGE_SHIFT == 0)
|
|
|
|
return 0;
|
2008-07-24 11:27:44 +07:00
|
|
|
|
2008-07-24 11:27:52 +07:00
|
|
|
if (!size_to_hstate(default_hstate_size)) {
|
|
|
|
default_hstate_size = HPAGE_SIZE;
|
|
|
|
if (!size_to_hstate(default_hstate_size))
|
|
|
|
hugetlb_add_hstate(HUGETLB_PAGE_ORDER);
|
2008-07-24 11:27:44 +07:00
|
|
|
}
|
2008-07-24 11:27:52 +07:00
|
|
|
default_hstate_idx = size_to_hstate(default_hstate_size) - hstates;
|
|
|
|
if (default_hstate_max_huge_pages)
|
|
|
|
default_hstate.max_huge_pages = default_hstate_max_huge_pages;
|
2008-07-24 11:27:44 +07:00
|
|
|
|
|
|
|
hugetlb_init_hstates();
|
|
|
|
|
2008-07-24 11:27:47 +07:00
|
|
|
gather_bootmem_prealloc();
|
|
|
|
|
2008-07-24 11:27:44 +07:00
|
|
|
report_hugepages();
|
|
|
|
|
|
|
|
hugetlb_sysfs_init();
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
module_init(hugetlb_init);
|
|
|
|
|
|
|
|
/* Should be called on processing a hugepagesz=... option */
|
|
|
|
void __init hugetlb_add_hstate(unsigned order)
|
|
|
|
{
|
|
|
|
struct hstate *h;
|
2008-07-24 11:27:48 +07:00
|
|
|
unsigned long i;
|
|
|
|
|
2008-07-24 11:27:44 +07:00
|
|
|
if (size_to_hstate(PAGE_SIZE << order)) {
|
|
|
|
printk(KERN_WARNING "hugepagesz= specified twice, ignoring\n");
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
BUG_ON(max_hstate >= HUGE_MAX_HSTATE);
|
|
|
|
BUG_ON(order == 0);
|
|
|
|
h = &hstates[max_hstate++];
|
|
|
|
h->order = order;
|
|
|
|
h->mask = ~((1ULL << (order + PAGE_SHIFT)) - 1);
|
2008-07-24 11:27:48 +07:00
|
|
|
h->nr_huge_pages = 0;
|
|
|
|
h->free_huge_pages = 0;
|
|
|
|
for (i = 0; i < MAX_NUMNODES; ++i)
|
|
|
|
INIT_LIST_HEAD(&h->hugepage_freelists[i]);
|
|
|
|
h->hugetlb_next_nid = first_node(node_online_map);
|
2008-07-24 11:27:44 +07:00
|
|
|
snprintf(h->name, HSTATE_NAME_LEN, "hugepages-%lukB",
|
|
|
|
huge_page_size(h)/1024);
|
2008-07-24 11:27:48 +07:00
|
|
|
|
2008-07-24 11:27:44 +07:00
|
|
|
parsed_hstate = h;
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:52 +07:00
|
|
|
static int __init hugetlb_nrpages_setup(char *s)
|
2008-07-24 11:27:44 +07:00
|
|
|
{
|
|
|
|
unsigned long *mhp;
|
2008-07-24 11:27:48 +07:00
|
|
|
static unsigned long *last_mhp;
|
2008-07-24 11:27:44 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* !max_hstate means we haven't parsed a hugepagesz= parameter yet,
|
|
|
|
* so this hugepages= parameter goes to the "default hstate".
|
|
|
|
*/
|
|
|
|
if (!max_hstate)
|
|
|
|
mhp = &default_hstate_max_huge_pages;
|
|
|
|
else
|
|
|
|
mhp = &parsed_hstate->max_huge_pages;
|
|
|
|
|
2008-07-24 11:27:48 +07:00
|
|
|
if (mhp == last_mhp) {
|
|
|
|
printk(KERN_WARNING "hugepages= specified twice without "
|
|
|
|
"interleaving hugepagesz=, ignoring\n");
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:44 +07:00
|
|
|
if (sscanf(s, "%lu", mhp) <= 0)
|
|
|
|
*mhp = 0;
|
|
|
|
|
2008-07-24 11:27:48 +07:00
|
|
|
/*
|
|
|
|
* Global state is always initialized later in hugetlb_init.
|
|
|
|
* But we need to allocate >= MAX_ORDER hstates here early to still
|
|
|
|
* use the bootmem allocator.
|
|
|
|
*/
|
|
|
|
if (max_hstate && parsed_hstate->order >= MAX_ORDER)
|
|
|
|
hugetlb_hstate_alloc_pages(parsed_hstate);
|
|
|
|
|
|
|
|
last_mhp = mhp;
|
|
|
|
|
2008-07-24 11:27:44 +07:00
|
|
|
return 1;
|
|
|
|
}
|
2008-07-24 11:27:52 +07:00
|
|
|
__setup("hugepages=", hugetlb_nrpages_setup);
|
|
|
|
|
|
|
|
static int __init hugetlb_default_setup(char *s)
|
|
|
|
{
|
|
|
|
default_hstate_size = memparse(s, &s);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
__setup("default_hugepagesz=", hugetlb_default_setup);
|
2008-07-24 11:27:44 +07:00
|
|
|
|
2008-07-26 09:44:37 +07:00
|
|
|
static unsigned int cpuset_mems_nr(unsigned int *array)
|
|
|
|
{
|
|
|
|
int node;
|
|
|
|
unsigned int nr = 0;
|
|
|
|
|
|
|
|
for_each_node_mask(node, cpuset_current_mems_allowed)
|
|
|
|
nr += array[node];
|
|
|
|
|
|
|
|
return nr;
|
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef CONFIG_SYSCTL
|
2005-04-17 05:20:36 +07:00
|
|
|
int hugetlb_sysctl_handler(struct ctl_table *table, int write,
|
|
|
|
struct file *file, void __user *buffer,
|
|
|
|
size_t *length, loff_t *ppos)
|
|
|
|
{
|
2008-07-24 11:27:42 +07:00
|
|
|
struct hstate *h = &default_hstate;
|
|
|
|
unsigned long tmp;
|
|
|
|
|
|
|
|
if (!write)
|
|
|
|
tmp = h->max_huge_pages;
|
|
|
|
|
|
|
|
table->data = &tmp;
|
|
|
|
table->maxlen = sizeof(unsigned long);
|
2005-04-17 05:20:36 +07:00
|
|
|
proc_doulongvec_minmax(table, write, file, buffer, length, ppos);
|
2008-07-24 11:27:42 +07:00
|
|
|
|
|
|
|
if (write)
|
|
|
|
h->max_huge_pages = set_max_huge_pages(h, tmp);
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
return 0;
|
|
|
|
}
|
2007-07-17 18:03:13 +07:00
|
|
|
|
|
|
|
int hugetlb_treat_movable_handler(struct ctl_table *table, int write,
|
|
|
|
struct file *file, void __user *buffer,
|
|
|
|
size_t *length, loff_t *ppos)
|
|
|
|
{
|
|
|
|
proc_dointvec(table, write, file, buffer, length, ppos);
|
|
|
|
if (hugepages_treat_as_movable)
|
|
|
|
htlb_alloc_mask = GFP_HIGHUSER_MOVABLE;
|
|
|
|
else
|
|
|
|
htlb_alloc_mask = GFP_HIGHUSER;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2008-02-08 19:18:18 +07:00
|
|
|
int hugetlb_overcommit_handler(struct ctl_table *table, int write,
|
|
|
|
struct file *file, void __user *buffer,
|
|
|
|
size_t *length, loff_t *ppos)
|
|
|
|
{
|
2008-07-24 11:27:41 +07:00
|
|
|
struct hstate *h = &default_hstate;
|
2008-07-24 11:27:42 +07:00
|
|
|
unsigned long tmp;
|
|
|
|
|
|
|
|
if (!write)
|
|
|
|
tmp = h->nr_overcommit_huge_pages;
|
|
|
|
|
|
|
|
table->data = &tmp;
|
|
|
|
table->maxlen = sizeof(unsigned long);
|
2008-02-08 19:18:18 +07:00
|
|
|
proc_doulongvec_minmax(table, write, file, buffer, length, ppos);
|
2008-07-24 11:27:42 +07:00
|
|
|
|
|
|
|
if (write) {
|
|
|
|
spin_lock(&hugetlb_lock);
|
|
|
|
h->nr_overcommit_huge_pages = tmp;
|
|
|
|
spin_unlock(&hugetlb_lock);
|
|
|
|
}
|
|
|
|
|
2008-02-08 19:18:18 +07:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
#endif /* CONFIG_SYSCTL */
|
|
|
|
|
2008-10-16 02:50:22 +07:00
|
|
|
void hugetlb_report_meminfo(struct seq_file *m)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2008-07-24 11:27:41 +07:00
|
|
|
struct hstate *h = &default_hstate;
|
2008-10-16 02:50:22 +07:00
|
|
|
seq_printf(m,
|
2008-10-19 10:26:32 +07:00
|
|
|
"HugePages_Total: %5lu\n"
|
|
|
|
"HugePages_Free: %5lu\n"
|
|
|
|
"HugePages_Rsvd: %5lu\n"
|
|
|
|
"HugePages_Surp: %5lu\n"
|
|
|
|
"Hugepagesize: %8lu kB\n",
|
2008-07-24 11:27:41 +07:00
|
|
|
h->nr_huge_pages,
|
|
|
|
h->free_huge_pages,
|
|
|
|
h->resv_huge_pages,
|
|
|
|
h->surplus_huge_pages,
|
|
|
|
1UL << (huge_page_order(h) + PAGE_SHIFT - 10));
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
int hugetlb_report_node_meminfo(int nid, char *buf)
|
|
|
|
{
|
2008-07-24 11:27:41 +07:00
|
|
|
struct hstate *h = &default_hstate;
|
2005-04-17 05:20:36 +07:00
|
|
|
return sprintf(buf,
|
|
|
|
"Node %d HugePages_Total: %5u\n"
|
2008-03-27 04:37:53 +07:00
|
|
|
"Node %d HugePages_Free: %5u\n"
|
|
|
|
"Node %d HugePages_Surp: %5u\n",
|
2008-07-24 11:27:41 +07:00
|
|
|
nid, h->nr_huge_pages_node[nid],
|
|
|
|
nid, h->free_huge_pages_node[nid],
|
|
|
|
nid, h->surplus_huge_pages_node[nid]);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Return the number pages of memory we physically have, in PAGE_SIZE units. */
|
|
|
|
unsigned long hugetlb_total_pages(void)
|
|
|
|
{
|
2008-07-24 11:27:41 +07:00
|
|
|
struct hstate *h = &default_hstate;
|
|
|
|
return h->nr_huge_pages * pages_per_huge_page(h);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
static int hugetlb_acct_memory(struct hstate *h, long delta)
|
2008-07-24 11:27:22 +07:00
|
|
|
{
|
|
|
|
int ret = -ENOMEM;
|
|
|
|
|
|
|
|
spin_lock(&hugetlb_lock);
|
|
|
|
/*
|
|
|
|
* When cpuset is configured, it breaks the strict hugetlb page
|
|
|
|
* reservation as the accounting is done on a global variable. Such
|
|
|
|
* reservation is completely rubbish in the presence of cpuset because
|
|
|
|
* the reservation is not checked against page availability for the
|
|
|
|
* current cpuset. Application can still potentially OOM'ed by kernel
|
|
|
|
* with lack of free htlb page in cpuset that the task is in.
|
|
|
|
* Attempt to enforce strict accounting with cpuset is almost
|
|
|
|
* impossible (or too ugly) because cpuset is too fluid that
|
|
|
|
* task or memory node can be dynamically moved between cpusets.
|
|
|
|
*
|
|
|
|
* The change of semantics for shared hugetlb mapping with cpuset is
|
|
|
|
* undesirable. However, in order to preserve some of the semantics,
|
|
|
|
* we fall back to check against current free page availability as
|
|
|
|
* a best attempt and hopefully to minimize the impact of changing
|
|
|
|
* semantics that cpuset has.
|
|
|
|
*/
|
|
|
|
if (delta > 0) {
|
2008-07-24 11:27:41 +07:00
|
|
|
if (gather_surplus_pages(h, delta) < 0)
|
2008-07-24 11:27:22 +07:00
|
|
|
goto out;
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
if (delta > cpuset_mems_nr(h->free_huge_pages_node)) {
|
|
|
|
return_unused_surplus_pages(h, delta);
|
2008-07-24 11:27:22 +07:00
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
ret = 0;
|
|
|
|
if (delta < 0)
|
2008-07-24 11:27:41 +07:00
|
|
|
return_unused_surplus_pages(h, (unsigned long) -delta);
|
2008-07-24 11:27:22 +07:00
|
|
|
|
|
|
|
out:
|
|
|
|
spin_unlock(&hugetlb_lock);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:32 +07:00
|
|
|
static void hugetlb_vm_op_open(struct vm_area_struct *vma)
|
|
|
|
{
|
|
|
|
struct resv_map *reservations = vma_resv_map(vma);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This new VMA should share its siblings reservation map if present.
|
|
|
|
* The VMA will only ever have a valid reservation map pointer where
|
|
|
|
* it is being copied for another still existing VMA. As that VMA
|
|
|
|
* has a reference to the reservation map it cannot dissappear until
|
|
|
|
* after this open call completes. It is therefore safe to take a
|
|
|
|
* new reference here without additional locking.
|
|
|
|
*/
|
|
|
|
if (reservations)
|
|
|
|
kref_get(&reservations->refs);
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:23 +07:00
|
|
|
static void hugetlb_vm_op_close(struct vm_area_struct *vma)
|
|
|
|
{
|
2008-07-24 11:27:41 +07:00
|
|
|
struct hstate *h = hstate_vma(vma);
|
2008-07-24 11:27:32 +07:00
|
|
|
struct resv_map *reservations = vma_resv_map(vma);
|
|
|
|
unsigned long reserve;
|
|
|
|
unsigned long start;
|
|
|
|
unsigned long end;
|
|
|
|
|
|
|
|
if (reservations) {
|
2008-07-24 11:27:41 +07:00
|
|
|
start = vma_hugecache_offset(h, vma, vma->vm_start);
|
|
|
|
end = vma_hugecache_offset(h, vma, vma->vm_end);
|
2008-07-24 11:27:32 +07:00
|
|
|
|
|
|
|
reserve = (end - start) -
|
|
|
|
region_count(&reservations->regions, start, end);
|
|
|
|
|
|
|
|
kref_put(&reservations->refs, resv_map_release);
|
|
|
|
|
2008-07-24 11:27:59 +07:00
|
|
|
if (reserve) {
|
2008-07-24 11:27:41 +07:00
|
|
|
hugetlb_acct_memory(h, -reserve);
|
2008-07-24 11:27:59 +07:00
|
|
|
hugetlb_put_quota(vma->vm_file->f_mapping, reserve);
|
|
|
|
}
|
2008-07-24 11:27:32 +07:00
|
|
|
}
|
2008-07-24 11:27:23 +07:00
|
|
|
}
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* We cannot handle pagefaults against hugetlb pages at all. They cause
|
|
|
|
* handle_mm_fault() to try to instantiate regular-sized pages in the
|
|
|
|
* hugegpage VMA. do_page_fault() is supposed to trap this, so BUG is we get
|
|
|
|
* this far.
|
|
|
|
*/
|
2007-07-19 15:47:03 +07:00
|
|
|
static int hugetlb_vm_op_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
BUG();
|
2007-07-19 15:47:03 +07:00
|
|
|
return 0;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
struct vm_operations_struct hugetlb_vm_ops = {
|
2007-07-19 15:47:03 +07:00
|
|
|
.fault = hugetlb_vm_op_fault,
|
2008-07-24 11:27:32 +07:00
|
|
|
.open = hugetlb_vm_op_open,
|
2008-07-24 11:27:23 +07:00
|
|
|
.close = hugetlb_vm_op_close,
|
2005-04-17 05:20:36 +07:00
|
|
|
};
|
|
|
|
|
2006-01-06 15:10:44 +07:00
|
|
|
static pte_t make_huge_pte(struct vm_area_struct *vma, struct page *page,
|
|
|
|
int writable)
|
2005-06-22 07:14:44 +07:00
|
|
|
{
|
|
|
|
pte_t entry;
|
|
|
|
|
2006-01-06 15:10:44 +07:00
|
|
|
if (writable) {
|
2005-06-22 07:14:44 +07:00
|
|
|
entry =
|
|
|
|
pte_mkwrite(pte_mkdirty(mk_pte(page, vma->vm_page_prot)));
|
|
|
|
} else {
|
2008-04-28 16:13:29 +07:00
|
|
|
entry = huge_pte_wrprotect(mk_pte(page, vma->vm_page_prot));
|
2005-06-22 07:14:44 +07:00
|
|
|
}
|
|
|
|
entry = pte_mkyoung(entry);
|
|
|
|
entry = pte_mkhuge(entry);
|
|
|
|
|
|
|
|
return entry;
|
|
|
|
}
|
|
|
|
|
2006-01-06 15:10:44 +07:00
|
|
|
static void set_huge_ptep_writable(struct vm_area_struct *vma,
|
|
|
|
unsigned long address, pte_t *ptep)
|
|
|
|
{
|
|
|
|
pte_t entry;
|
|
|
|
|
2008-04-28 16:13:29 +07:00
|
|
|
entry = pte_mkwrite(pte_mkdirty(huge_ptep_get(ptep)));
|
|
|
|
if (huge_ptep_set_access_flags(vma, address, ptep, entry, 1)) {
|
2007-06-17 00:16:12 +07:00
|
|
|
update_mmu_cache(vma, address, entry);
|
|
|
|
}
|
2006-01-06 15:10:44 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
|
2005-06-22 07:14:44 +07:00
|
|
|
int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
|
|
|
|
struct vm_area_struct *vma)
|
|
|
|
{
|
|
|
|
pte_t *src_pte, *dst_pte, entry;
|
|
|
|
struct page *ptepage;
|
2005-10-20 11:23:43 +07:00
|
|
|
unsigned long addr;
|
2006-01-06 15:10:44 +07:00
|
|
|
int cow;
|
2008-07-24 11:27:41 +07:00
|
|
|
struct hstate *h = hstate_vma(vma);
|
|
|
|
unsigned long sz = huge_page_size(h);
|
2006-01-06 15:10:44 +07:00
|
|
|
|
|
|
|
cow = (vma->vm_flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE;
|
2005-06-22 07:14:44 +07:00
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
for (addr = vma->vm_start; addr < vma->vm_end; addr += sz) {
|
2005-10-30 08:16:23 +07:00
|
|
|
src_pte = huge_pte_offset(src, addr);
|
|
|
|
if (!src_pte)
|
|
|
|
continue;
|
2008-07-24 11:27:41 +07:00
|
|
|
dst_pte = huge_pte_alloc(dst, addr, sz);
|
2005-06-22 07:14:44 +07:00
|
|
|
if (!dst_pte)
|
|
|
|
goto nomem;
|
2008-01-24 20:49:25 +07:00
|
|
|
|
|
|
|
/* If the pagetables are shared don't copy or take references */
|
|
|
|
if (dst_pte == src_pte)
|
|
|
|
continue;
|
|
|
|
|
2005-10-30 08:16:23 +07:00
|
|
|
spin_lock(&dst->page_table_lock);
|
2008-06-06 12:45:57 +07:00
|
|
|
spin_lock_nested(&src->page_table_lock, SINGLE_DEPTH_NESTING);
|
2008-04-28 16:13:29 +07:00
|
|
|
if (!huge_pte_none(huge_ptep_get(src_pte))) {
|
2006-01-06 15:10:44 +07:00
|
|
|
if (cow)
|
2008-04-28 16:13:29 +07:00
|
|
|
huge_ptep_set_wrprotect(src, addr, src_pte);
|
|
|
|
entry = huge_ptep_get(src_pte);
|
2005-10-20 11:23:43 +07:00
|
|
|
ptepage = pte_page(entry);
|
|
|
|
get_page(ptepage);
|
|
|
|
set_huge_pte_at(dst, addr, dst_pte, entry);
|
|
|
|
}
|
|
|
|
spin_unlock(&src->page_table_lock);
|
2005-10-30 08:16:23 +07:00
|
|
|
spin_unlock(&dst->page_table_lock);
|
2005-06-22 07:14:44 +07:00
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
nomem:
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
|
2006-10-11 15:20:46 +07:00
|
|
|
void __unmap_hugepage_range(struct vm_area_struct *vma, unsigned long start,
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
unsigned long end, struct page *ref_page)
|
2005-06-22 07:14:44 +07:00
|
|
|
{
|
|
|
|
struct mm_struct *mm = vma->vm_mm;
|
|
|
|
unsigned long address;
|
2005-08-06 01:59:35 +07:00
|
|
|
pte_t *ptep;
|
2005-06-22 07:14:44 +07:00
|
|
|
pte_t pte;
|
|
|
|
struct page *page;
|
2006-10-04 16:15:24 +07:00
|
|
|
struct page *tmp;
|
2008-07-24 11:27:41 +07:00
|
|
|
struct hstate *h = hstate_vma(vma);
|
|
|
|
unsigned long sz = huge_page_size(h);
|
|
|
|
|
2006-12-07 11:31:39 +07:00
|
|
|
/*
|
|
|
|
* A page gathering list, protected by per file i_mmap_lock. The
|
|
|
|
* lock is used to avoid list corruption from multiple unmapping
|
|
|
|
* of the same page since we are using page->lru.
|
|
|
|
*/
|
2006-10-04 16:15:24 +07:00
|
|
|
LIST_HEAD(page_list);
|
2005-06-22 07:14:44 +07:00
|
|
|
|
|
|
|
WARN_ON(!is_vm_hugetlb_page(vma));
|
2008-07-24 11:27:41 +07:00
|
|
|
BUG_ON(start & ~huge_page_mask(h));
|
|
|
|
BUG_ON(end & ~huge_page_mask(h));
|
2005-06-22 07:14:44 +07:00
|
|
|
|
mmu-notifiers: core
With KVM/GFP/XPMEM there isn't just the primary CPU MMU pointing to pages.
There are secondary MMUs (with secondary sptes and secondary tlbs) too.
sptes in the kvm case are shadow pagetables, but when I say spte in
mmu-notifier context, I mean "secondary pte". In GRU case there's no
actual secondary pte and there's only a secondary tlb because the GRU
secondary MMU has no knowledge about sptes and every secondary tlb miss
event in the MMU always generates a page fault that has to be resolved by
the CPU (this is not the case of KVM where the a secondary tlb miss will
walk sptes in hardware and it will refill the secondary tlb transparently
to software if the corresponding spte is present). The same way
zap_page_range has to invalidate the pte before freeing the page, the spte
(and secondary tlb) must also be invalidated before any page is freed and
reused.
Currently we take a page_count pin on every page mapped by sptes, but that
means the pages can't be swapped whenever they're mapped by any spte
because they're part of the guest working set. Furthermore a spte unmap
event can immediately lead to a page to be freed when the pin is released
(so requiring the same complex and relatively slow tlb_gather smp safe
logic we have in zap_page_range and that can be avoided completely if the
spte unmap event doesn't require an unpin of the page previously mapped in
the secondary MMU).
The mmu notifiers allow kvm/GRU/XPMEM to attach to the tsk->mm and know
when the VM is swapping or freeing or doing anything on the primary MMU so
that the secondary MMU code can drop sptes before the pages are freed,
avoiding all page pinning and allowing 100% reliable swapping of guest
physical address space. Furthermore it avoids the code that teardown the
mappings of the secondary MMU, to implement a logic like tlb_gather in
zap_page_range that would require many IPI to flush other cpu tlbs, for
each fixed number of spte unmapped.
To make an example: if what happens on the primary MMU is a protection
downgrade (from writeable to wrprotect) the secondary MMU mappings will be
invalidated, and the next secondary-mmu-page-fault will call
get_user_pages and trigger a do_wp_page through get_user_pages if it
called get_user_pages with write=1, and it'll re-establishing an updated
spte or secondary-tlb-mapping on the copied page. Or it will setup a
readonly spte or readonly tlb mapping if it's a guest-read, if it calls
get_user_pages with write=0. This is just an example.
This allows to map any page pointed by any pte (and in turn visible in the
primary CPU MMU), into a secondary MMU (be it a pure tlb like GRU, or an
full MMU with both sptes and secondary-tlb like the shadow-pagetable layer
with kvm), or a remote DMA in software like XPMEM (hence needing of
schedule in XPMEM code to send the invalidate to the remote node, while no
need to schedule in kvm/gru as it's an immediate event like invalidating
primary-mmu pte).
At least for KVM without this patch it's impossible to swap guests
reliably. And having this feature and removing the page pin allows
several other optimizations that simplify life considerably.
Dependencies:
1) mm_take_all_locks() to register the mmu notifier when the whole VM
isn't doing anything with "mm". This allows mmu notifier users to keep
track if the VM is in the middle of the invalidate_range_begin/end
critical section with an atomic counter incraese in range_begin and
decreased in range_end. No secondary MMU page fault is allowed to map
any spte or secondary tlb reference, while the VM is in the middle of
range_begin/end as any page returned by get_user_pages in that critical
section could later immediately be freed without any further
->invalidate_page notification (invalidate_range_begin/end works on
ranges and ->invalidate_page isn't called immediately before freeing
the page). To stop all page freeing and pagetable overwrites the
mmap_sem must be taken in write mode and all other anon_vma/i_mmap
locks must be taken too.
2) It'd be a waste to add branches in the VM if nobody could possibly
run KVM/GRU/XPMEM on the kernel, so mmu notifiers will only enabled if
CONFIG_KVM=m/y. In the current kernel kvm won't yet take advantage of
mmu notifiers, but this already allows to compile a KVM external module
against a kernel with mmu notifiers enabled and from the next pull from
kvm.git we'll start using them. And GRU/XPMEM will also be able to
continue the development by enabling KVM=m in their config, until they
submit all GRU/XPMEM GPLv2 code to the mainline kernel. Then they can
also enable MMU_NOTIFIERS in the same way KVM does it (even if KVM=n).
This guarantees nobody selects MMU_NOTIFIER=y if KVM and GRU and XPMEM
are all =n.
The mmu_notifier_register call can fail because mm_take_all_locks may be
interrupted by a signal and return -EINTR. Because mmu_notifier_reigster
is used when a driver startup, a failure can be gracefully handled. Here
an example of the change applied to kvm to register the mmu notifiers.
Usually when a driver startups other allocations are required anyway and
-ENOMEM failure paths exists already.
struct kvm *kvm_arch_create_vm(void)
{
struct kvm *kvm = kzalloc(sizeof(struct kvm), GFP_KERNEL);
+ int err;
if (!kvm)
return ERR_PTR(-ENOMEM);
INIT_LIST_HEAD(&kvm->arch.active_mmu_pages);
+ kvm->arch.mmu_notifier.ops = &kvm_mmu_notifier_ops;
+ err = mmu_notifier_register(&kvm->arch.mmu_notifier, current->mm);
+ if (err) {
+ kfree(kvm);
+ return ERR_PTR(err);
+ }
+
return kvm;
}
mmu_notifier_unregister returns void and it's reliable.
The patch also adds a few needed but missing includes that would prevent
kernel to compile after these changes on non-x86 archs (x86 didn't need
them by luck).
[akpm@linux-foundation.org: coding-style fixes]
[akpm@linux-foundation.org: fix mm/filemap_xip.c build]
[akpm@linux-foundation.org: fix mm/mmu_notifier.c build]
Signed-off-by: Andrea Arcangeli <andrea@qumranet.com>
Signed-off-by: Nick Piggin <npiggin@suse.de>
Signed-off-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Jack Steiner <steiner@sgi.com>
Cc: Robin Holt <holt@sgi.com>
Cc: Nick Piggin <npiggin@suse.de>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Kanoj Sarcar <kanojsarcar@yahoo.com>
Cc: Roland Dreier <rdreier@cisco.com>
Cc: Steve Wise <swise@opengridcomputing.com>
Cc: Avi Kivity <avi@qumranet.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Anthony Liguori <aliguori@us.ibm.com>
Cc: Chris Wright <chrisw@redhat.com>
Cc: Marcelo Tosatti <marcelo@kvack.org>
Cc: Eric Dumazet <dada1@cosmosbay.com>
Cc: "Paul E. McKenney" <paulmck@us.ibm.com>
Cc: Izik Eidus <izike@qumranet.com>
Cc: Anthony Liguori <aliguori@us.ibm.com>
Cc: Rik van Riel <riel@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-29 05:46:29 +07:00
|
|
|
mmu_notifier_invalidate_range_start(mm, start, end);
|
2005-10-30 08:16:30 +07:00
|
|
|
spin_lock(&mm->page_table_lock);
|
2008-07-24 11:27:41 +07:00
|
|
|
for (address = start; address < end; address += sz) {
|
2005-08-06 01:59:35 +07:00
|
|
|
ptep = huge_pte_offset(mm, address);
|
2005-10-30 08:16:46 +07:00
|
|
|
if (!ptep)
|
2005-08-06 01:59:35 +07:00
|
|
|
continue;
|
|
|
|
|
2006-12-07 11:32:03 +07:00
|
|
|
if (huge_pmd_unshare(mm, &address, ptep))
|
|
|
|
continue;
|
|
|
|
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
/*
|
|
|
|
* If a reference page is supplied, it is because a specific
|
|
|
|
* page is being unmapped, not a range. Ensure the page we
|
|
|
|
* are about to unmap is the actual page of interest.
|
|
|
|
*/
|
|
|
|
if (ref_page) {
|
|
|
|
pte = huge_ptep_get(ptep);
|
|
|
|
if (huge_pte_none(pte))
|
|
|
|
continue;
|
|
|
|
page = pte_page(pte);
|
|
|
|
if (page != ref_page)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Mark the VMA as having unmapped its page so that
|
|
|
|
* future faults in this VMA will fail rather than
|
|
|
|
* looking like data was lost
|
|
|
|
*/
|
|
|
|
set_vma_resv_flags(vma, HPAGE_RESV_UNMAPPED);
|
|
|
|
}
|
|
|
|
|
2005-08-06 01:59:35 +07:00
|
|
|
pte = huge_ptep_get_and_clear(mm, address, ptep);
|
2008-04-28 16:13:29 +07:00
|
|
|
if (huge_pte_none(pte))
|
2005-06-22 07:14:44 +07:00
|
|
|
continue;
|
2005-08-06 01:59:35 +07:00
|
|
|
|
2005-06-22 07:14:44 +07:00
|
|
|
page = pte_page(pte);
|
2007-02-09 05:20:27 +07:00
|
|
|
if (pte_dirty(pte))
|
|
|
|
set_page_dirty(page);
|
2006-10-04 16:15:24 +07:00
|
|
|
list_add(&page->lru, &page_list);
|
2005-06-22 07:14:44 +07:00
|
|
|
}
|
2005-04-17 05:20:36 +07:00
|
|
|
spin_unlock(&mm->page_table_lock);
|
2005-10-30 08:16:30 +07:00
|
|
|
flush_tlb_range(vma, start, end);
|
mmu-notifiers: core
With KVM/GFP/XPMEM there isn't just the primary CPU MMU pointing to pages.
There are secondary MMUs (with secondary sptes and secondary tlbs) too.
sptes in the kvm case are shadow pagetables, but when I say spte in
mmu-notifier context, I mean "secondary pte". In GRU case there's no
actual secondary pte and there's only a secondary tlb because the GRU
secondary MMU has no knowledge about sptes and every secondary tlb miss
event in the MMU always generates a page fault that has to be resolved by
the CPU (this is not the case of KVM where the a secondary tlb miss will
walk sptes in hardware and it will refill the secondary tlb transparently
to software if the corresponding spte is present). The same way
zap_page_range has to invalidate the pte before freeing the page, the spte
(and secondary tlb) must also be invalidated before any page is freed and
reused.
Currently we take a page_count pin on every page mapped by sptes, but that
means the pages can't be swapped whenever they're mapped by any spte
because they're part of the guest working set. Furthermore a spte unmap
event can immediately lead to a page to be freed when the pin is released
(so requiring the same complex and relatively slow tlb_gather smp safe
logic we have in zap_page_range and that can be avoided completely if the
spte unmap event doesn't require an unpin of the page previously mapped in
the secondary MMU).
The mmu notifiers allow kvm/GRU/XPMEM to attach to the tsk->mm and know
when the VM is swapping or freeing or doing anything on the primary MMU so
that the secondary MMU code can drop sptes before the pages are freed,
avoiding all page pinning and allowing 100% reliable swapping of guest
physical address space. Furthermore it avoids the code that teardown the
mappings of the secondary MMU, to implement a logic like tlb_gather in
zap_page_range that would require many IPI to flush other cpu tlbs, for
each fixed number of spte unmapped.
To make an example: if what happens on the primary MMU is a protection
downgrade (from writeable to wrprotect) the secondary MMU mappings will be
invalidated, and the next secondary-mmu-page-fault will call
get_user_pages and trigger a do_wp_page through get_user_pages if it
called get_user_pages with write=1, and it'll re-establishing an updated
spte or secondary-tlb-mapping on the copied page. Or it will setup a
readonly spte or readonly tlb mapping if it's a guest-read, if it calls
get_user_pages with write=0. This is just an example.
This allows to map any page pointed by any pte (and in turn visible in the
primary CPU MMU), into a secondary MMU (be it a pure tlb like GRU, or an
full MMU with both sptes and secondary-tlb like the shadow-pagetable layer
with kvm), or a remote DMA in software like XPMEM (hence needing of
schedule in XPMEM code to send the invalidate to the remote node, while no
need to schedule in kvm/gru as it's an immediate event like invalidating
primary-mmu pte).
At least for KVM without this patch it's impossible to swap guests
reliably. And having this feature and removing the page pin allows
several other optimizations that simplify life considerably.
Dependencies:
1) mm_take_all_locks() to register the mmu notifier when the whole VM
isn't doing anything with "mm". This allows mmu notifier users to keep
track if the VM is in the middle of the invalidate_range_begin/end
critical section with an atomic counter incraese in range_begin and
decreased in range_end. No secondary MMU page fault is allowed to map
any spte or secondary tlb reference, while the VM is in the middle of
range_begin/end as any page returned by get_user_pages in that critical
section could later immediately be freed without any further
->invalidate_page notification (invalidate_range_begin/end works on
ranges and ->invalidate_page isn't called immediately before freeing
the page). To stop all page freeing and pagetable overwrites the
mmap_sem must be taken in write mode and all other anon_vma/i_mmap
locks must be taken too.
2) It'd be a waste to add branches in the VM if nobody could possibly
run KVM/GRU/XPMEM on the kernel, so mmu notifiers will only enabled if
CONFIG_KVM=m/y. In the current kernel kvm won't yet take advantage of
mmu notifiers, but this already allows to compile a KVM external module
against a kernel with mmu notifiers enabled and from the next pull from
kvm.git we'll start using them. And GRU/XPMEM will also be able to
continue the development by enabling KVM=m in their config, until they
submit all GRU/XPMEM GPLv2 code to the mainline kernel. Then they can
also enable MMU_NOTIFIERS in the same way KVM does it (even if KVM=n).
This guarantees nobody selects MMU_NOTIFIER=y if KVM and GRU and XPMEM
are all =n.
The mmu_notifier_register call can fail because mm_take_all_locks may be
interrupted by a signal and return -EINTR. Because mmu_notifier_reigster
is used when a driver startup, a failure can be gracefully handled. Here
an example of the change applied to kvm to register the mmu notifiers.
Usually when a driver startups other allocations are required anyway and
-ENOMEM failure paths exists already.
struct kvm *kvm_arch_create_vm(void)
{
struct kvm *kvm = kzalloc(sizeof(struct kvm), GFP_KERNEL);
+ int err;
if (!kvm)
return ERR_PTR(-ENOMEM);
INIT_LIST_HEAD(&kvm->arch.active_mmu_pages);
+ kvm->arch.mmu_notifier.ops = &kvm_mmu_notifier_ops;
+ err = mmu_notifier_register(&kvm->arch.mmu_notifier, current->mm);
+ if (err) {
+ kfree(kvm);
+ return ERR_PTR(err);
+ }
+
return kvm;
}
mmu_notifier_unregister returns void and it's reliable.
The patch also adds a few needed but missing includes that would prevent
kernel to compile after these changes on non-x86 archs (x86 didn't need
them by luck).
[akpm@linux-foundation.org: coding-style fixes]
[akpm@linux-foundation.org: fix mm/filemap_xip.c build]
[akpm@linux-foundation.org: fix mm/mmu_notifier.c build]
Signed-off-by: Andrea Arcangeli <andrea@qumranet.com>
Signed-off-by: Nick Piggin <npiggin@suse.de>
Signed-off-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Jack Steiner <steiner@sgi.com>
Cc: Robin Holt <holt@sgi.com>
Cc: Nick Piggin <npiggin@suse.de>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Kanoj Sarcar <kanojsarcar@yahoo.com>
Cc: Roland Dreier <rdreier@cisco.com>
Cc: Steve Wise <swise@opengridcomputing.com>
Cc: Avi Kivity <avi@qumranet.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Anthony Liguori <aliguori@us.ibm.com>
Cc: Chris Wright <chrisw@redhat.com>
Cc: Marcelo Tosatti <marcelo@kvack.org>
Cc: Eric Dumazet <dada1@cosmosbay.com>
Cc: "Paul E. McKenney" <paulmck@us.ibm.com>
Cc: Izik Eidus <izike@qumranet.com>
Cc: Anthony Liguori <aliguori@us.ibm.com>
Cc: Rik van Riel <riel@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-29 05:46:29 +07:00
|
|
|
mmu_notifier_invalidate_range_end(mm, start, end);
|
2006-10-04 16:15:24 +07:00
|
|
|
list_for_each_entry_safe(page, tmp, &page_list, lru) {
|
|
|
|
list_del(&page->lru);
|
|
|
|
put_page(page);
|
|
|
|
}
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
2005-06-22 07:14:44 +07:00
|
|
|
|
2006-10-11 15:20:46 +07:00
|
|
|
void unmap_hugepage_range(struct vm_area_struct *vma, unsigned long start,
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
unsigned long end, struct page *ref_page)
|
2006-10-11 15:20:46 +07:00
|
|
|
{
|
2008-07-24 11:27:43 +07:00
|
|
|
spin_lock(&vma->vm_file->f_mapping->i_mmap_lock);
|
|
|
|
__unmap_hugepage_range(vma, start, end, ref_page);
|
|
|
|
spin_unlock(&vma->vm_file->f_mapping->i_mmap_lock);
|
2006-10-11 15:20:46 +07:00
|
|
|
}
|
|
|
|
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
/*
|
|
|
|
* This is called when the original mapper is failing to COW a MAP_PRIVATE
|
|
|
|
* mappping it owns the reserve page for. The intention is to unmap the page
|
|
|
|
* from other VMAs and let the children be SIGKILLed if they are faulting the
|
|
|
|
* same region.
|
|
|
|
*/
|
2008-10-19 10:27:06 +07:00
|
|
|
static int unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma,
|
|
|
|
struct page *page, unsigned long address)
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
{
|
2008-11-13 04:24:56 +07:00
|
|
|
struct hstate *h = hstate_vma(vma);
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
struct vm_area_struct *iter_vma;
|
|
|
|
struct address_space *mapping;
|
|
|
|
struct prio_tree_iter iter;
|
|
|
|
pgoff_t pgoff;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* vm_pgoff is in PAGE_SIZE units, hence the different calculation
|
|
|
|
* from page cache lookup which is in HPAGE_SIZE units.
|
|
|
|
*/
|
2008-11-13 04:24:56 +07:00
|
|
|
address = address & huge_page_mask(h);
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
pgoff = ((address - vma->vm_start) >> PAGE_SHIFT)
|
|
|
|
+ (vma->vm_pgoff >> PAGE_SHIFT);
|
|
|
|
mapping = (struct address_space *)page_private(page);
|
|
|
|
|
|
|
|
vma_prio_tree_foreach(iter_vma, &iter, &mapping->i_mmap, pgoff, pgoff) {
|
|
|
|
/* Do not unmap the current VMA */
|
|
|
|
if (iter_vma == vma)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Unmap the page from other VMAs without their own reserves.
|
|
|
|
* They get marked to be SIGKILLed if they fault in these
|
|
|
|
* areas. This is because a future no-page fault on this VMA
|
|
|
|
* could insert a zeroed page instead of the data existing
|
|
|
|
* from the time of fork. This would look like data corruption
|
|
|
|
*/
|
|
|
|
if (!is_vma_resv_set(iter_vma, HPAGE_RESV_OWNER))
|
|
|
|
unmap_hugepage_range(iter_vma,
|
2008-11-13 04:24:56 +07:00
|
|
|
address, address + huge_page_size(h),
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
page);
|
|
|
|
}
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2006-01-06 15:10:44 +07:00
|
|
|
static int hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma,
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
unsigned long address, pte_t *ptep, pte_t pte,
|
|
|
|
struct page *pagecache_page)
|
2006-01-06 15:10:44 +07:00
|
|
|
{
|
2008-07-24 11:27:41 +07:00
|
|
|
struct hstate *h = hstate_vma(vma);
|
2006-01-06 15:10:44 +07:00
|
|
|
struct page *old_page, *new_page;
|
2006-03-22 15:08:51 +07:00
|
|
|
int avoidcopy;
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
int outside_reserve = 0;
|
2006-01-06 15:10:44 +07:00
|
|
|
|
|
|
|
old_page = pte_page(pte);
|
|
|
|
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
retry_avoidcopy:
|
2006-01-06 15:10:44 +07:00
|
|
|
/* If no-one else is actually using this page, avoid the copy
|
|
|
|
* and just make the page writable */
|
|
|
|
avoidcopy = (page_count(old_page) == 1);
|
|
|
|
if (avoidcopy) {
|
|
|
|
set_huge_ptep_writable(vma, address, ptep);
|
2007-07-19 15:47:05 +07:00
|
|
|
return 0;
|
2006-01-06 15:10:44 +07:00
|
|
|
}
|
|
|
|
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
/*
|
|
|
|
* If the process that created a MAP_PRIVATE mapping is about to
|
|
|
|
* perform a COW due to a shared page count, attempt to satisfy
|
|
|
|
* the allocation without using the existing reserves. The pagecache
|
|
|
|
* page is used to determine if the reserve at this address was
|
|
|
|
* consumed or not. If reserves were used, a partial faulted mapping
|
|
|
|
* at the time of fork() could consume its reserves on COW instead
|
|
|
|
* of the full address range.
|
|
|
|
*/
|
2009-05-29 04:34:40 +07:00
|
|
|
if (!(vma->vm_flags & VM_MAYSHARE) &&
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
is_vma_resv_set(vma, HPAGE_RESV_OWNER) &&
|
|
|
|
old_page != pagecache_page)
|
|
|
|
outside_reserve = 1;
|
|
|
|
|
2006-01-06 15:10:44 +07:00
|
|
|
page_cache_get(old_page);
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
new_page = alloc_huge_page(vma, address, outside_reserve);
|
2006-01-06 15:10:44 +07:00
|
|
|
|
2007-11-15 07:59:39 +07:00
|
|
|
if (IS_ERR(new_page)) {
|
2006-01-06 15:10:44 +07:00
|
|
|
page_cache_release(old_page);
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If a process owning a MAP_PRIVATE mapping fails to COW,
|
|
|
|
* it is due to references held by a child and an insufficient
|
|
|
|
* huge page pool. To guarantee the original mappers
|
|
|
|
* reliability, unmap the page from child processes. The child
|
|
|
|
* may get SIGKILLed if it later faults.
|
|
|
|
*/
|
|
|
|
if (outside_reserve) {
|
|
|
|
BUG_ON(huge_pte_none(pte));
|
|
|
|
if (unmap_ref_private(mm, vma, old_page, address)) {
|
|
|
|
BUG_ON(page_count(old_page) != 1);
|
|
|
|
BUG_ON(huge_pte_none(pte));
|
|
|
|
goto retry_avoidcopy;
|
|
|
|
}
|
|
|
|
WARN_ON_ONCE(1);
|
|
|
|
}
|
|
|
|
|
2007-11-15 07:59:39 +07:00
|
|
|
return -PTR_ERR(new_page);
|
2006-01-06 15:10:44 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
spin_unlock(&mm->page_table_lock);
|
2006-12-13 00:14:55 +07:00
|
|
|
copy_huge_page(new_page, old_page, address, vma);
|
mm: fix PageUptodate data race
After running SetPageUptodate, preceeding stores to the page contents to
actually bring it uptodate may not be ordered with the store to set the
page uptodate.
Therefore, another CPU which checks PageUptodate is true, then reads the
page contents can get stale data.
Fix this by having an smp_wmb before SetPageUptodate, and smp_rmb after
PageUptodate.
Many places that test PageUptodate, do so with the page locked, and this
would be enough to ensure memory ordering in those places if
SetPageUptodate were only called while the page is locked. Unfortunately
that is not always the case for some filesystems, but it could be an idea
for the future.
Also bring the handling of anonymous page uptodateness in line with that of
file backed page management, by marking anon pages as uptodate when they
_are_ uptodate, rather than when our implementation requires that they be
marked as such. Doing allows us to get rid of the smp_wmb's in the page
copying functions, which were especially added for anonymous pages for an
analogous memory ordering problem. Both file and anonymous pages are
handled with the same barriers.
FAQ:
Q. Why not do this in flush_dcache_page?
A. Firstly, flush_dcache_page handles only one side (the smb side) of the
ordering protocol; we'd still need smp_rmb somewhere. Secondly, hiding away
memory barriers in a completely unrelated function is nasty; at least in the
PageUptodate macros, they are located together with (half) the operations
involved in the ordering. Thirdly, the smp_wmb is only required when first
bringing the page uptodate, wheras flush_dcache_page should be called each time
it is written to through the kernel mapping. It is logically the wrong place to
put it.
Q. Why does this increase my text size / reduce my performance / etc.
A. Because it is adding the necessary instructions to eliminate the data-race.
Q. Can it be improved?
A. Yes, eg. if you were to create a rule that all SetPageUptodate operations
run under the page lock, we could avoid the smp_rmb places where PageUptodate
is queried under the page lock. Requires audit of all filesystems and at least
some would need reworking. That's great you're interested, I'm eagerly awaiting
your patches.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-02-05 13:29:34 +07:00
|
|
|
__SetPageUptodate(new_page);
|
2006-01-06 15:10:44 +07:00
|
|
|
spin_lock(&mm->page_table_lock);
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
ptep = huge_pte_offset(mm, address & huge_page_mask(h));
|
2008-04-28 16:13:29 +07:00
|
|
|
if (likely(pte_same(huge_ptep_get(ptep), pte))) {
|
2006-01-06 15:10:44 +07:00
|
|
|
/* Break COW */
|
2008-04-28 16:13:28 +07:00
|
|
|
huge_ptep_clear_flush(vma, address, ptep);
|
2006-01-06 15:10:44 +07:00
|
|
|
set_huge_pte_at(mm, address, ptep,
|
|
|
|
make_huge_pte(vma, new_page, 1));
|
|
|
|
/* Make the old page be freed below */
|
|
|
|
new_page = old_page;
|
|
|
|
}
|
|
|
|
page_cache_release(new_page);
|
|
|
|
page_cache_release(old_page);
|
2007-07-19 15:47:05 +07:00
|
|
|
return 0;
|
2006-01-06 15:10:44 +07:00
|
|
|
}
|
|
|
|
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
/* Return the pagecache page at a given address within a VMA */
|
2008-07-24 11:27:41 +07:00
|
|
|
static struct page *hugetlbfs_pagecache_page(struct hstate *h,
|
|
|
|
struct vm_area_struct *vma, unsigned long address)
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
{
|
|
|
|
struct address_space *mapping;
|
2008-07-24 11:27:26 +07:00
|
|
|
pgoff_t idx;
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
|
|
|
|
mapping = vma->vm_file->f_mapping;
|
2008-07-24 11:27:41 +07:00
|
|
|
idx = vma_hugecache_offset(h, vma, address);
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
|
|
|
|
return find_lock_page(mapping, idx);
|
|
|
|
}
|
|
|
|
|
2007-07-17 18:03:33 +07:00
|
|
|
static int hugetlb_no_page(struct mm_struct *mm, struct vm_area_struct *vma,
|
2009-06-23 19:49:05 +07:00
|
|
|
unsigned long address, pte_t *ptep, unsigned int flags)
|
2005-10-20 22:24:28 +07:00
|
|
|
{
|
2008-07-24 11:27:41 +07:00
|
|
|
struct hstate *h = hstate_vma(vma);
|
2005-10-20 22:24:28 +07:00
|
|
|
int ret = VM_FAULT_SIGBUS;
|
2008-07-24 11:27:26 +07:00
|
|
|
pgoff_t idx;
|
2005-10-30 08:16:46 +07:00
|
|
|
unsigned long size;
|
|
|
|
struct page *page;
|
|
|
|
struct address_space *mapping;
|
2006-01-06 15:10:44 +07:00
|
|
|
pte_t new_pte;
|
2005-10-30 08:16:46 +07:00
|
|
|
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
/*
|
|
|
|
* Currently, we are forced to kill the process in the event the
|
|
|
|
* original mapper has unmapped pages from the child due to a failed
|
|
|
|
* COW. Warn that such a situation has occured as it may not be obvious
|
|
|
|
*/
|
|
|
|
if (is_vma_resv_set(vma, HPAGE_RESV_UNMAPPED)) {
|
|
|
|
printk(KERN_WARNING
|
|
|
|
"PID %d killed due to inadequate hugepage pool\n",
|
|
|
|
current->pid);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2005-10-30 08:16:46 +07:00
|
|
|
mapping = vma->vm_file->f_mapping;
|
2008-07-24 11:27:41 +07:00
|
|
|
idx = vma_hugecache_offset(h, vma, address);
|
2005-10-30 08:16:46 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Use page lock to guard against racing truncation
|
|
|
|
* before we get page_table_lock.
|
|
|
|
*/
|
2006-01-06 15:10:49 +07:00
|
|
|
retry:
|
|
|
|
page = find_lock_page(mapping, idx);
|
|
|
|
if (!page) {
|
2008-07-24 11:27:41 +07:00
|
|
|
size = i_size_read(mapping->host) >> huge_page_shift(h);
|
2006-10-29 00:38:43 +07:00
|
|
|
if (idx >= size)
|
|
|
|
goto out;
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
page = alloc_huge_page(vma, address, 0);
|
2007-11-15 07:59:39 +07:00
|
|
|
if (IS_ERR(page)) {
|
|
|
|
ret = -PTR_ERR(page);
|
2006-01-06 15:10:49 +07:00
|
|
|
goto out;
|
|
|
|
}
|
2008-07-24 11:27:41 +07:00
|
|
|
clear_huge_page(page, address, huge_page_size(h));
|
mm: fix PageUptodate data race
After running SetPageUptodate, preceeding stores to the page contents to
actually bring it uptodate may not be ordered with the store to set the
page uptodate.
Therefore, another CPU which checks PageUptodate is true, then reads the
page contents can get stale data.
Fix this by having an smp_wmb before SetPageUptodate, and smp_rmb after
PageUptodate.
Many places that test PageUptodate, do so with the page locked, and this
would be enough to ensure memory ordering in those places if
SetPageUptodate were only called while the page is locked. Unfortunately
that is not always the case for some filesystems, but it could be an idea
for the future.
Also bring the handling of anonymous page uptodateness in line with that of
file backed page management, by marking anon pages as uptodate when they
_are_ uptodate, rather than when our implementation requires that they be
marked as such. Doing allows us to get rid of the smp_wmb's in the page
copying functions, which were especially added for anonymous pages for an
analogous memory ordering problem. Both file and anonymous pages are
handled with the same barriers.
FAQ:
Q. Why not do this in flush_dcache_page?
A. Firstly, flush_dcache_page handles only one side (the smb side) of the
ordering protocol; we'd still need smp_rmb somewhere. Secondly, hiding away
memory barriers in a completely unrelated function is nasty; at least in the
PageUptodate macros, they are located together with (half) the operations
involved in the ordering. Thirdly, the smp_wmb is only required when first
bringing the page uptodate, wheras flush_dcache_page should be called each time
it is written to through the kernel mapping. It is logically the wrong place to
put it.
Q. Why does this increase my text size / reduce my performance / etc.
A. Because it is adding the necessary instructions to eliminate the data-race.
Q. Can it be improved?
A. Yes, eg. if you were to create a rule that all SetPageUptodate operations
run under the page lock, we could avoid the smp_rmb places where PageUptodate
is queried under the page lock. Requires audit of all filesystems and at least
some would need reworking. That's great you're interested, I'm eagerly awaiting
your patches.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-02-05 13:29:34 +07:00
|
|
|
__SetPageUptodate(page);
|
2005-10-20 22:24:28 +07:00
|
|
|
|
2009-05-29 04:34:40 +07:00
|
|
|
if (vma->vm_flags & VM_MAYSHARE) {
|
2006-01-06 15:10:49 +07:00
|
|
|
int err;
|
2007-11-15 07:59:44 +07:00
|
|
|
struct inode *inode = mapping->host;
|
2006-01-06 15:10:49 +07:00
|
|
|
|
|
|
|
err = add_to_page_cache(page, mapping, idx, GFP_KERNEL);
|
|
|
|
if (err) {
|
|
|
|
put_page(page);
|
|
|
|
if (err == -EEXIST)
|
|
|
|
goto retry;
|
|
|
|
goto out;
|
|
|
|
}
|
2007-11-15 07:59:44 +07:00
|
|
|
|
|
|
|
spin_lock(&inode->i_lock);
|
2008-07-24 11:27:41 +07:00
|
|
|
inode->i_blocks += blocks_per_huge_page(h);
|
2007-11-15 07:59:44 +07:00
|
|
|
spin_unlock(&inode->i_lock);
|
2006-01-06 15:10:49 +07:00
|
|
|
} else
|
|
|
|
lock_page(page);
|
|
|
|
}
|
2006-01-06 15:10:44 +07:00
|
|
|
|
2008-08-13 05:08:47 +07:00
|
|
|
/*
|
|
|
|
* If we are going to COW a private mapping later, we examine the
|
|
|
|
* pending reservations for this page now. This will ensure that
|
|
|
|
* any allocations necessary to record that reservation occur outside
|
|
|
|
* the spinlock.
|
|
|
|
*/
|
2009-06-23 19:49:05 +07:00
|
|
|
if ((flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED))
|
2008-08-13 05:08:49 +07:00
|
|
|
if (vma_needs_reservation(h, vma, address) < 0) {
|
|
|
|
ret = VM_FAULT_OOM;
|
|
|
|
goto backout_unlocked;
|
|
|
|
}
|
2008-08-13 05:08:47 +07:00
|
|
|
|
2005-10-20 22:24:28 +07:00
|
|
|
spin_lock(&mm->page_table_lock);
|
2008-07-24 11:27:41 +07:00
|
|
|
size = i_size_read(mapping->host) >> huge_page_shift(h);
|
2005-10-30 08:16:46 +07:00
|
|
|
if (idx >= size)
|
|
|
|
goto backout;
|
|
|
|
|
2007-07-19 15:47:05 +07:00
|
|
|
ret = 0;
|
2008-04-28 16:13:29 +07:00
|
|
|
if (!huge_pte_none(huge_ptep_get(ptep)))
|
2005-10-30 08:16:46 +07:00
|
|
|
goto backout;
|
|
|
|
|
2006-01-06 15:10:44 +07:00
|
|
|
new_pte = make_huge_pte(vma, page, ((vma->vm_flags & VM_WRITE)
|
|
|
|
&& (vma->vm_flags & VM_SHARED)));
|
|
|
|
set_huge_pte_at(mm, address, ptep, new_pte);
|
|
|
|
|
2009-06-23 19:49:05 +07:00
|
|
|
if ((flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) {
|
2006-01-06 15:10:44 +07:00
|
|
|
/* Optimization, do the COW without a second fault */
|
hugetlb: guarantee that COW faults for a process that called mmap(MAP_PRIVATE) on hugetlbfs will succeed
After patch 2 in this series, a process that successfully calls mmap() for
a MAP_PRIVATE mapping will be guaranteed to successfully fault until a
process calls fork(). At that point, the next write fault from the parent
could fail due to COW if the child still has a reference.
We only reserve pages for the parent but a copy must be made to avoid
leaking data from the parent to the child after fork(). Reserves could be
taken for both parent and child at fork time to guarantee faults but if
the mapping is large it is highly likely we will not have sufficient pages
for the reservation, and it is common to fork only to exec() immediatly
after. A failure here would be very undesirable.
Note that the current behaviour of mainline with MAP_PRIVATE pages is
pretty bad. The following situation is allowed to occur today.
1. Process calls mmap(MAP_PRIVATE)
2. Process calls mlock() to fault all pages and makes sure it succeeds
3. Process forks()
4. Process writes to MAP_PRIVATE mapping while child still exists
5. If the COW fails at this point, the process gets SIGKILLed even though it
had taken care to ensure the pages existed
This patch improves the situation by guaranteeing the reliability of the
process that successfully calls mmap(). When the parent performs COW, it
will try to satisfy the allocation without using reserves. If that fails
the parent will steal the page leaving any children without a page.
Faults from the child after that point will result in failure. If the
child COW happens first, an attempt will be made to allocate the page
without reserves and the child will get SIGKILLed on failure.
To summarise the new behaviour:
1. If the original mapper performs COW on a private mapping with multiple
references, it will attempt to allocate a hugepage from the pool or
the buddy allocator without using the existing reserves. On fail, VMAs
mapping the same area are traversed and the page being COW'd is unmapped
where found. It will then steal the original page as the last mapper in
the normal way.
2. The VMAs the pages were unmapped from are flagged to note that pages
with data no longer exist. Future no-page faults on those VMAs will
terminate the process as otherwise it would appear that data was corrupted.
A warning is printed to the console that this situation occured.
2. If the child performs COW first, it will attempt to satisfy the COW
from the pool if there are enough pages or via the buddy allocator if
overcommit is allowed and the buddy allocator can satisfy the request. If
it fails, the child will be killed.
If the pool is large enough, existing applications will not notice that
the reserves were a factor. Existing applications depending on the
no-reserves been set are unlikely to exist as for much of the history of
hugetlbfs, pages were prefaulted at mmap(), allocating the pages at that
point or failing the mmap().
[npiggin@suse.de: fix CONFIG_HUGETLB=n build]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Acked-by: Adam Litke <agl@us.ibm.com>
Cc: Andy Whitcroft <apw@shadowen.org>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: Hugh Dickins <hugh@veritas.com>
Cc: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-07-24 11:27:25 +07:00
|
|
|
ret = hugetlb_cow(mm, vma, address, ptep, new_pte, page);
|
2006-01-06 15:10:44 +07:00
|
|
|
}
|
|
|
|
|
2005-10-20 22:24:28 +07:00
|
|
|
spin_unlock(&mm->page_table_lock);
|
2005-10-30 08:16:46 +07:00
|
|
|
unlock_page(page);
|
|
|
|
out:
|
2005-10-20 22:24:28 +07:00
|
|
|
return ret;
|
2005-10-30 08:16:46 +07:00
|
|
|
|
|
|
|
backout:
|
|
|
|
spin_unlock(&mm->page_table_lock);
|
2008-08-13 05:08:49 +07:00
|
|
|
backout_unlocked:
|
2005-10-30 08:16:46 +07:00
|
|
|
unlock_page(page);
|
|
|
|
put_page(page);
|
|
|
|
goto out;
|
2005-10-20 22:24:28 +07:00
|
|
|
}
|
|
|
|
|
2006-01-06 15:10:43 +07:00
|
|
|
int hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
|
2009-06-23 19:49:05 +07:00
|
|
|
unsigned long address, unsigned int flags)
|
2006-01-06 15:10:43 +07:00
|
|
|
{
|
|
|
|
pte_t *ptep;
|
|
|
|
pte_t entry;
|
2006-01-06 15:10:44 +07:00
|
|
|
int ret;
|
2008-08-13 05:08:47 +07:00
|
|
|
struct page *pagecache_page = NULL;
|
[PATCH] hugepage: serialize hugepage allocation and instantiation
Currently, no lock or mutex is held between allocating a hugepage and
inserting it into the pagetables / page cache. When we do go to insert the
page into pagetables or page cache, we recheck and may free the newly
allocated hugepage. However, since the number of hugepages in the system
is strictly limited, and it's usualy to want to use all of them, this can
still lead to spurious allocation failures.
For example, suppose two processes are both mapping (MAP_SHARED) the same
hugepage file, large enough to consume the entire available hugepage pool.
If they race instantiating the last page in the mapping, they will both
attempt to allocate the last available hugepage. One will fail, of course,
returning OOM from the fault and thus causing the process to be killed,
despite the fact that the entire mapping can, in fact, be instantiated.
The patch fixes this race by the simple method of adding a (sleeping) mutex
to serialize the hugepage fault path between allocation and insertion into
pagetables and/or page cache. It would be possible to avoid the
serialization by catching the allocation failures, waiting on some
condition, then rechecking to see if someone else has instantiated the page
for us. Given the likely frequency of hugepage instantiations, it seems
very doubtful it's worth the extra complexity.
This patch causes no regression on the libhugetlbfs testsuite, and one
test, which can trigger this race now passes where it previously failed.
Actually, the test still sometimes fails, though less often and only as a
shmat() failure, rather processes getting OOM killed by the VM. The dodgy
heuristic tests in fs/hugetlbfs/inode.c for whether there's enough hugepage
space aren't protected by the new mutex, and would be ugly to do so, so
there's still a race there. Another patch to replace those tests with
something saner for this reason as well as others coming...
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-03-22 15:08:53 +07:00
|
|
|
static DEFINE_MUTEX(hugetlb_instantiation_mutex);
|
2008-07-24 11:27:41 +07:00
|
|
|
struct hstate *h = hstate_vma(vma);
|
2006-01-06 15:10:43 +07:00
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
ptep = huge_pte_alloc(mm, address, huge_page_size(h));
|
2006-01-06 15:10:43 +07:00
|
|
|
if (!ptep)
|
|
|
|
return VM_FAULT_OOM;
|
|
|
|
|
[PATCH] hugepage: serialize hugepage allocation and instantiation
Currently, no lock or mutex is held between allocating a hugepage and
inserting it into the pagetables / page cache. When we do go to insert the
page into pagetables or page cache, we recheck and may free the newly
allocated hugepage. However, since the number of hugepages in the system
is strictly limited, and it's usualy to want to use all of them, this can
still lead to spurious allocation failures.
For example, suppose two processes are both mapping (MAP_SHARED) the same
hugepage file, large enough to consume the entire available hugepage pool.
If they race instantiating the last page in the mapping, they will both
attempt to allocate the last available hugepage. One will fail, of course,
returning OOM from the fault and thus causing the process to be killed,
despite the fact that the entire mapping can, in fact, be instantiated.
The patch fixes this race by the simple method of adding a (sleeping) mutex
to serialize the hugepage fault path between allocation and insertion into
pagetables and/or page cache. It would be possible to avoid the
serialization by catching the allocation failures, waiting on some
condition, then rechecking to see if someone else has instantiated the page
for us. Given the likely frequency of hugepage instantiations, it seems
very doubtful it's worth the extra complexity.
This patch causes no regression on the libhugetlbfs testsuite, and one
test, which can trigger this race now passes where it previously failed.
Actually, the test still sometimes fails, though less often and only as a
shmat() failure, rather processes getting OOM killed by the VM. The dodgy
heuristic tests in fs/hugetlbfs/inode.c for whether there's enough hugepage
space aren't protected by the new mutex, and would be ugly to do so, so
there's still a race there. Another patch to replace those tests with
something saner for this reason as well as others coming...
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-03-22 15:08:53 +07:00
|
|
|
/*
|
|
|
|
* Serialize hugepage allocation and instantiation, so that we don't
|
|
|
|
* get spurious allocation failures if two CPUs race to instantiate
|
|
|
|
* the same page in the page cache.
|
|
|
|
*/
|
|
|
|
mutex_lock(&hugetlb_instantiation_mutex);
|
2008-04-28 16:13:29 +07:00
|
|
|
entry = huge_ptep_get(ptep);
|
|
|
|
if (huge_pte_none(entry)) {
|
2009-06-23 19:49:05 +07:00
|
|
|
ret = hugetlb_no_page(mm, vma, address, ptep, flags);
|
2008-10-16 12:01:11 +07:00
|
|
|
goto out_mutex;
|
[PATCH] hugepage: serialize hugepage allocation and instantiation
Currently, no lock or mutex is held between allocating a hugepage and
inserting it into the pagetables / page cache. When we do go to insert the
page into pagetables or page cache, we recheck and may free the newly
allocated hugepage. However, since the number of hugepages in the system
is strictly limited, and it's usualy to want to use all of them, this can
still lead to spurious allocation failures.
For example, suppose two processes are both mapping (MAP_SHARED) the same
hugepage file, large enough to consume the entire available hugepage pool.
If they race instantiating the last page in the mapping, they will both
attempt to allocate the last available hugepage. One will fail, of course,
returning OOM from the fault and thus causing the process to be killed,
despite the fact that the entire mapping can, in fact, be instantiated.
The patch fixes this race by the simple method of adding a (sleeping) mutex
to serialize the hugepage fault path between allocation and insertion into
pagetables and/or page cache. It would be possible to avoid the
serialization by catching the allocation failures, waiting on some
condition, then rechecking to see if someone else has instantiated the page
for us. Given the likely frequency of hugepage instantiations, it seems
very doubtful it's worth the extra complexity.
This patch causes no regression on the libhugetlbfs testsuite, and one
test, which can trigger this race now passes where it previously failed.
Actually, the test still sometimes fails, though less often and only as a
shmat() failure, rather processes getting OOM killed by the VM. The dodgy
heuristic tests in fs/hugetlbfs/inode.c for whether there's enough hugepage
space aren't protected by the new mutex, and would be ugly to do so, so
there's still a race there. Another patch to replace those tests with
something saner for this reason as well as others coming...
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-03-22 15:08:53 +07:00
|
|
|
}
|
2006-01-06 15:10:43 +07:00
|
|
|
|
2007-07-19 15:47:05 +07:00
|
|
|
ret = 0;
|
2006-01-06 15:10:44 +07:00
|
|
|
|
2008-08-13 05:08:47 +07:00
|
|
|
/*
|
|
|
|
* If we are going to COW the mapping later, we examine the pending
|
|
|
|
* reservations for this page now. This will ensure that any
|
|
|
|
* allocations necessary to record that reservation occur outside the
|
|
|
|
* spinlock. For private mappings, we also lookup the pagecache
|
|
|
|
* page now as it is used to determine if a reservation has been
|
|
|
|
* consumed.
|
|
|
|
*/
|
2009-06-23 19:49:05 +07:00
|
|
|
if ((flags & FAULT_FLAG_WRITE) && !pte_write(entry)) {
|
2008-08-13 05:08:49 +07:00
|
|
|
if (vma_needs_reservation(h, vma, address) < 0) {
|
|
|
|
ret = VM_FAULT_OOM;
|
2008-10-16 12:01:11 +07:00
|
|
|
goto out_mutex;
|
2008-08-13 05:08:49 +07:00
|
|
|
}
|
2008-08-13 05:08:47 +07:00
|
|
|
|
2009-05-29 04:34:40 +07:00
|
|
|
if (!(vma->vm_flags & VM_MAYSHARE))
|
2008-08-13 05:08:47 +07:00
|
|
|
pagecache_page = hugetlbfs_pagecache_page(h,
|
|
|
|
vma, address);
|
|
|
|
}
|
|
|
|
|
2006-01-06 15:10:44 +07:00
|
|
|
spin_lock(&mm->page_table_lock);
|
|
|
|
/* Check for a racing update before calling hugetlb_cow */
|
2008-10-16 12:01:11 +07:00
|
|
|
if (unlikely(!pte_same(entry, huge_ptep_get(ptep))))
|
|
|
|
goto out_page_table_lock;
|
|
|
|
|
|
|
|
|
2009-06-23 19:49:05 +07:00
|
|
|
if (flags & FAULT_FLAG_WRITE) {
|
2008-10-16 12:01:11 +07:00
|
|
|
if (!pte_write(entry)) {
|
2008-08-13 05:08:47 +07:00
|
|
|
ret = hugetlb_cow(mm, vma, address, ptep, entry,
|
|
|
|
pagecache_page);
|
2008-10-16 12:01:11 +07:00
|
|
|
goto out_page_table_lock;
|
|
|
|
}
|
|
|
|
entry = pte_mkdirty(entry);
|
|
|
|
}
|
|
|
|
entry = pte_mkyoung(entry);
|
2009-06-23 19:49:05 +07:00
|
|
|
if (huge_ptep_set_access_flags(vma, address, ptep, entry,
|
|
|
|
flags & FAULT_FLAG_WRITE))
|
2008-10-16 12:01:11 +07:00
|
|
|
update_mmu_cache(vma, address, entry);
|
|
|
|
|
|
|
|
out_page_table_lock:
|
2006-01-06 15:10:44 +07:00
|
|
|
spin_unlock(&mm->page_table_lock);
|
2008-08-13 05:08:47 +07:00
|
|
|
|
|
|
|
if (pagecache_page) {
|
|
|
|
unlock_page(pagecache_page);
|
|
|
|
put_page(pagecache_page);
|
|
|
|
}
|
|
|
|
|
2008-10-16 12:01:11 +07:00
|
|
|
out_mutex:
|
[PATCH] hugepage: serialize hugepage allocation and instantiation
Currently, no lock or mutex is held between allocating a hugepage and
inserting it into the pagetables / page cache. When we do go to insert the
page into pagetables or page cache, we recheck and may free the newly
allocated hugepage. However, since the number of hugepages in the system
is strictly limited, and it's usualy to want to use all of them, this can
still lead to spurious allocation failures.
For example, suppose two processes are both mapping (MAP_SHARED) the same
hugepage file, large enough to consume the entire available hugepage pool.
If they race instantiating the last page in the mapping, they will both
attempt to allocate the last available hugepage. One will fail, of course,
returning OOM from the fault and thus causing the process to be killed,
despite the fact that the entire mapping can, in fact, be instantiated.
The patch fixes this race by the simple method of adding a (sleeping) mutex
to serialize the hugepage fault path between allocation and insertion into
pagetables and/or page cache. It would be possible to avoid the
serialization by catching the allocation failures, waiting on some
condition, then rechecking to see if someone else has instantiated the page
for us. Given the likely frequency of hugepage instantiations, it seems
very doubtful it's worth the extra complexity.
This patch causes no regression on the libhugetlbfs testsuite, and one
test, which can trigger this race now passes where it previously failed.
Actually, the test still sometimes fails, though less often and only as a
shmat() failure, rather processes getting OOM killed by the VM. The dodgy
heuristic tests in fs/hugetlbfs/inode.c for whether there's enough hugepage
space aren't protected by the new mutex, and would be ugly to do so, so
there's still a race there. Another patch to replace those tests with
something saner for this reason as well as others coming...
Signed-off-by: David Gibson <dwg@au1.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-03-22 15:08:53 +07:00
|
|
|
mutex_unlock(&hugetlb_instantiation_mutex);
|
2006-01-06 15:10:44 +07:00
|
|
|
|
|
|
|
return ret;
|
2006-01-06 15:10:43 +07:00
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:50 +07:00
|
|
|
/* Can be overriden by architectures */
|
|
|
|
__attribute__((weak)) struct page *
|
|
|
|
follow_huge_pud(struct mm_struct *mm, unsigned long address,
|
|
|
|
pud_t *pud, int write)
|
|
|
|
{
|
|
|
|
BUG();
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2008-10-19 10:27:10 +07:00
|
|
|
static int huge_zeropage_ok(pte_t *ptep, int write, int shared)
|
|
|
|
{
|
|
|
|
if (!ptep || write || shared)
|
|
|
|
return 0;
|
|
|
|
else
|
|
|
|
return huge_pte_none(huge_ptep_get(ptep));
|
|
|
|
}
|
|
|
|
|
2005-06-22 07:14:44 +07:00
|
|
|
int follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma,
|
|
|
|
struct page **pages, struct vm_area_struct **vmas,
|
2007-11-15 07:59:33 +07:00
|
|
|
unsigned long *position, int *length, int i,
|
|
|
|
int write)
|
2005-06-22 07:14:44 +07:00
|
|
|
{
|
2006-03-22 15:09:03 +07:00
|
|
|
unsigned long pfn_offset;
|
|
|
|
unsigned long vaddr = *position;
|
2005-06-22 07:14:44 +07:00
|
|
|
int remainder = *length;
|
2008-07-24 11:27:41 +07:00
|
|
|
struct hstate *h = hstate_vma(vma);
|
2008-10-19 10:27:10 +07:00
|
|
|
int zeropage_ok = 0;
|
|
|
|
int shared = vma->vm_flags & VM_SHARED;
|
2005-06-22 07:14:44 +07:00
|
|
|
|
2005-10-20 11:23:43 +07:00
|
|
|
spin_lock(&mm->page_table_lock);
|
2005-06-22 07:14:44 +07:00
|
|
|
while (vaddr < vma->vm_end && remainder) {
|
2005-10-30 08:16:46 +07:00
|
|
|
pte_t *pte;
|
|
|
|
struct page *page;
|
2005-06-22 07:14:44 +07:00
|
|
|
|
2005-10-30 08:16:46 +07:00
|
|
|
/*
|
|
|
|
* Some archs (sparc64, sh*) have multiple pte_ts to
|
|
|
|
* each hugepage. We have to make * sure we get the
|
|
|
|
* first, for the page indexing below to work.
|
|
|
|
*/
|
2008-07-24 11:27:41 +07:00
|
|
|
pte = huge_pte_offset(mm, vaddr & huge_page_mask(h));
|
2008-10-19 10:27:10 +07:00
|
|
|
if (huge_zeropage_ok(pte, write, shared))
|
|
|
|
zeropage_ok = 1;
|
2005-06-22 07:14:44 +07:00
|
|
|
|
2008-10-19 10:27:10 +07:00
|
|
|
if (!pte ||
|
|
|
|
(huge_pte_none(huge_ptep_get(pte)) && !zeropage_ok) ||
|
2008-04-28 16:13:29 +07:00
|
|
|
(write && !pte_write(huge_ptep_get(pte)))) {
|
2005-10-30 08:16:46 +07:00
|
|
|
int ret;
|
2005-06-22 07:14:44 +07:00
|
|
|
|
2005-10-30 08:16:46 +07:00
|
|
|
spin_unlock(&mm->page_table_lock);
|
2007-11-15 07:59:33 +07:00
|
|
|
ret = hugetlb_fault(mm, vma, vaddr, write);
|
2005-10-30 08:16:46 +07:00
|
|
|
spin_lock(&mm->page_table_lock);
|
2007-08-23 04:01:51 +07:00
|
|
|
if (!(ret & VM_FAULT_ERROR))
|
2005-10-30 08:16:46 +07:00
|
|
|
continue;
|
2005-06-22 07:14:44 +07:00
|
|
|
|
2005-10-30 08:16:46 +07:00
|
|
|
remainder = 0;
|
|
|
|
if (!i)
|
|
|
|
i = -EFAULT;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:41 +07:00
|
|
|
pfn_offset = (vaddr & ~huge_page_mask(h)) >> PAGE_SHIFT;
|
2008-04-28 16:13:29 +07:00
|
|
|
page = pte_page(huge_ptep_get(pte));
|
2006-03-22 15:09:03 +07:00
|
|
|
same_page:
|
2006-03-31 17:29:57 +07:00
|
|
|
if (pages) {
|
2008-10-19 10:27:10 +07:00
|
|
|
if (zeropage_ok)
|
|
|
|
pages[i] = ZERO_PAGE(0);
|
|
|
|
else
|
2008-11-07 03:53:26 +07:00
|
|
|
pages[i] = mem_map_offset(page, pfn_offset);
|
2008-10-19 10:27:10 +07:00
|
|
|
get_page(pages[i]);
|
2006-03-31 17:29:57 +07:00
|
|
|
}
|
2005-06-22 07:14:44 +07:00
|
|
|
|
|
|
|
if (vmas)
|
|
|
|
vmas[i] = vma;
|
|
|
|
|
|
|
|
vaddr += PAGE_SIZE;
|
2006-03-22 15:09:03 +07:00
|
|
|
++pfn_offset;
|
2005-06-22 07:14:44 +07:00
|
|
|
--remainder;
|
|
|
|
++i;
|
2006-03-22 15:09:03 +07:00
|
|
|
if (vaddr < vma->vm_end && remainder &&
|
2008-07-24 11:27:41 +07:00
|
|
|
pfn_offset < pages_per_huge_page(h)) {
|
2006-03-22 15:09:03 +07:00
|
|
|
/*
|
|
|
|
* We use pfn_offset to avoid touching the pageframes
|
|
|
|
* of this compound page.
|
|
|
|
*/
|
|
|
|
goto same_page;
|
|
|
|
}
|
2005-06-22 07:14:44 +07:00
|
|
|
}
|
2005-10-20 11:23:43 +07:00
|
|
|
spin_unlock(&mm->page_table_lock);
|
2005-06-22 07:14:44 +07:00
|
|
|
*length = remainder;
|
|
|
|
*position = vaddr;
|
|
|
|
|
|
|
|
return i;
|
|
|
|
}
|
2006-03-22 15:08:50 +07:00
|
|
|
|
|
|
|
void hugetlb_change_protection(struct vm_area_struct *vma,
|
|
|
|
unsigned long address, unsigned long end, pgprot_t newprot)
|
|
|
|
{
|
|
|
|
struct mm_struct *mm = vma->vm_mm;
|
|
|
|
unsigned long start = address;
|
|
|
|
pte_t *ptep;
|
|
|
|
pte_t pte;
|
2008-07-24 11:27:41 +07:00
|
|
|
struct hstate *h = hstate_vma(vma);
|
2006-03-22 15:08:50 +07:00
|
|
|
|
|
|
|
BUG_ON(address >= end);
|
|
|
|
flush_cache_range(vma, address, end);
|
|
|
|
|
2006-12-07 11:32:03 +07:00
|
|
|
spin_lock(&vma->vm_file->f_mapping->i_mmap_lock);
|
2006-03-22 15:08:50 +07:00
|
|
|
spin_lock(&mm->page_table_lock);
|
2008-07-24 11:27:41 +07:00
|
|
|
for (; address < end; address += huge_page_size(h)) {
|
2006-03-22 15:08:50 +07:00
|
|
|
ptep = huge_pte_offset(mm, address);
|
|
|
|
if (!ptep)
|
|
|
|
continue;
|
2006-12-07 11:32:03 +07:00
|
|
|
if (huge_pmd_unshare(mm, &address, ptep))
|
|
|
|
continue;
|
2008-04-28 16:13:29 +07:00
|
|
|
if (!huge_pte_none(huge_ptep_get(ptep))) {
|
2006-03-22 15:08:50 +07:00
|
|
|
pte = huge_ptep_get_and_clear(mm, address, ptep);
|
|
|
|
pte = pte_mkhuge(pte_modify(pte, newprot));
|
|
|
|
set_huge_pte_at(mm, address, ptep, pte);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
spin_unlock(&mm->page_table_lock);
|
2006-12-07 11:32:03 +07:00
|
|
|
spin_unlock(&vma->vm_file->f_mapping->i_mmap_lock);
|
2006-03-22 15:08:50 +07:00
|
|
|
|
|
|
|
flush_tlb_range(vma, start, end);
|
|
|
|
}
|
|
|
|
|
2008-07-24 11:27:23 +07:00
|
|
|
int hugetlb_reserve_pages(struct inode *inode,
|
|
|
|
long from, long to,
|
2009-02-10 21:02:27 +07:00
|
|
|
struct vm_area_struct *vma,
|
|
|
|
int acctflag)
|
2007-10-16 15:26:19 +07:00
|
|
|
{
|
2009-02-11 23:34:16 +07:00
|
|
|
long ret, chg;
|
2008-07-24 11:27:41 +07:00
|
|
|
struct hstate *h = hstate_inode(inode);
|
2007-10-16 15:26:19 +07:00
|
|
|
|
2009-02-11 23:34:16 +07:00
|
|
|
/*
|
|
|
|
* Only apply hugepage reservation if asked. At fault time, an
|
|
|
|
* attempt will be made for VM_NORESERVE to allocate a page
|
|
|
|
* and filesystem quota without using reserves
|
|
|
|
*/
|
|
|
|
if (acctflag & VM_NORESERVE)
|
|
|
|
return 0;
|
|
|
|
|
2008-07-24 11:27:23 +07:00
|
|
|
/*
|
|
|
|
* Shared mappings base their reservation on the number of pages that
|
|
|
|
* are already allocated on behalf of the file. Private mappings need
|
|
|
|
* to reserve the full area even if read-only as mprotect() may be
|
|
|
|
* called to make the mapping read-write. Assume !vma is a shm mapping
|
|
|
|
*/
|
2009-05-29 04:34:40 +07:00
|
|
|
if (!vma || vma->vm_flags & VM_MAYSHARE)
|
2008-07-24 11:27:23 +07:00
|
|
|
chg = region_chg(&inode->i_mapping->private_list, from, to);
|
2009-02-11 23:34:16 +07:00
|
|
|
else {
|
|
|
|
struct resv_map *resv_map = resv_map_alloc();
|
|
|
|
if (!resv_map)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
2008-07-24 11:27:23 +07:00
|
|
|
chg = to - from;
|
2008-07-24 11:27:32 +07:00
|
|
|
|
2009-02-11 23:34:16 +07:00
|
|
|
set_vma_resv_map(vma, resv_map);
|
|
|
|
set_vma_resv_flags(vma, HPAGE_RESV_OWNER);
|
|
|
|
}
|
|
|
|
|
2007-10-16 15:26:19 +07:00
|
|
|
if (chg < 0)
|
|
|
|
return chg;
|
2007-05-09 16:33:34 +07:00
|
|
|
|
2009-02-11 23:34:16 +07:00
|
|
|
/* There must be enough filesystem quota for the mapping */
|
2007-11-15 07:59:42 +07:00
|
|
|
if (hugetlb_get_quota(inode->i_mapping, chg))
|
|
|
|
return -ENOSPC;
|
2009-02-10 21:02:27 +07:00
|
|
|
|
|
|
|
/*
|
2009-02-11 23:34:16 +07:00
|
|
|
* Check enough hugepages are available for the reservation.
|
|
|
|
* Hand back the quota if there are not
|
2009-02-10 21:02:27 +07:00
|
|
|
*/
|
2008-07-24 11:27:41 +07:00
|
|
|
ret = hugetlb_acct_memory(h, chg);
|
2008-01-14 15:55:19 +07:00
|
|
|
if (ret < 0) {
|
|
|
|
hugetlb_put_quota(inode->i_mapping, chg);
|
2006-06-23 16:03:15 +07:00
|
|
|
return ret;
|
2008-01-14 15:55:19 +07:00
|
|
|
}
|
2009-02-11 23:34:16 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Account for the reservations made. Shared mappings record regions
|
|
|
|
* that have reservations as they are shared by multiple VMAs.
|
|
|
|
* When the last VMA disappears, the region map says how much
|
|
|
|
* the reservation was and the page cache tells how much of
|
|
|
|
* the reservation was consumed. Private mappings are per-VMA and
|
|
|
|
* only the consumed reservations are tracked. When the VMA
|
|
|
|
* disappears, the original reservation is the VMA size and the
|
|
|
|
* consumed reservations are stored in the map. Hence, nothing
|
|
|
|
* else has to be done for private mappings here
|
|
|
|
*/
|
2009-05-29 04:34:40 +07:00
|
|
|
if (!vma || vma->vm_flags & VM_MAYSHARE)
|
2008-07-24 11:27:23 +07:00
|
|
|
region_add(&inode->i_mapping->private_list, from, to);
|
2006-06-23 16:03:15 +07:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
void hugetlb_unreserve_pages(struct inode *inode, long offset, long freed)
|
|
|
|
{
|
2008-07-24 11:27:41 +07:00
|
|
|
struct hstate *h = hstate_inode(inode);
|
2006-06-23 16:03:15 +07:00
|
|
|
long chg = region_truncate(&inode->i_mapping->private_list, offset);
|
2007-11-15 07:59:44 +07:00
|
|
|
|
|
|
|
spin_lock(&inode->i_lock);
|
2009-07-30 05:02:16 +07:00
|
|
|
inode->i_blocks -= (blocks_per_huge_page(h) * freed);
|
2007-11-15 07:59:44 +07:00
|
|
|
spin_unlock(&inode->i_lock);
|
|
|
|
|
2007-11-15 07:59:42 +07:00
|
|
|
hugetlb_put_quota(inode->i_mapping, (chg - freed));
|
2008-07-24 11:27:41 +07:00
|
|
|
hugetlb_acct_memory(h, -(chg - freed));
|
2006-06-23 16:03:15 +07:00
|
|
|
}
|