2015-09-05 05:46:04 +07:00
|
|
|
/*
|
|
|
|
* include/linux/userfaultfd.h
|
|
|
|
*
|
|
|
|
* Copyright (C) 2007 Davide Libenzi <davidel@xmailserver.org>
|
|
|
|
* Copyright (C) 2015 Red Hat, Inc.
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
|
|
|
|
#ifndef _LINUX_USERFAULTFD_H
|
|
|
|
#define _LINUX_USERFAULTFD_H
|
|
|
|
|
|
|
|
#include <linux/types.h>
|
|
|
|
|
2017-02-23 06:42:06 +07:00
|
|
|
/*
|
|
|
|
* If the UFFDIO_API is upgraded someday, the UFFDIO_UNREGISTER and
|
|
|
|
* UFFDIO_WAKE ioctls should be defined as _IOW and not as _IOR. In
|
|
|
|
* userfaultfd.h we assumed the kernel was reading (instead _IOC_READ
|
|
|
|
* means the userland is reading).
|
|
|
|
*/
|
2015-09-05 05:46:04 +07:00
|
|
|
#define UFFD_API ((__u64)0xAA)
|
userfaultfd: non-cooperative: rollback userfaultfd_exit
Patch series "userfaultfd non-cooperative further update for 4.11 merge
window".
Unfortunately I noticed one relevant bug in userfaultfd_exit while doing
more testing. I've been doing testing before and this was also tested
by kbuild bot and exercised by the selftest, but this bug never
reproduced before.
I dropped userfaultfd_exit as result. I dropped it because of
implementation difficulty in receiving signals in __mmput and because I
think -ENOSPC as result from the background UFFDIO_COPY should be enough
already.
Before I decided to remove userfaultfd_exit, I noticed userfaultfd_exit
wasn't exercised by the selftest and when I tried to exercise it, after
moving it to a more correct place in __mmput where it would make more
sense and where the vma list is stable, it resulted in the
event_wait_completion in D state. So then I added the second patch to
be sure even if we call userfaultfd_event_wait_completion too late
during task exit(), we won't risk to generate tasks in D state. The
same check exists in handle_userfault() for the same reason, except it
makes a difference there, while here is just a robustness check and it's
run under WARN_ON_ONCE.
While looking at the userfaultfd_event_wait_completion() function I
looked back at its callers too while at it and I think it's not ok to
stop executing dup_fctx on the fcs list because we relay on
userfaultfd_event_wait_completion to execute
userfaultfd_ctx_put(fctx->orig) which is paired against
userfaultfd_ctx_get(fctx->orig) in dup_userfault just before
list_add(fcs). This change only takes care of fctx->orig but this area
also needs further review looking for similar problems in fctx->new.
The only patch that is urgent is the first because it's an use after
free during a SMP race condition that affects all processes if
CONFIG_USERFAULTFD=y. Very hard to reproduce though and probably
impossible without SLUB poisoning enabled.
This patch (of 3):
I once reproduced this oops with the userfaultfd selftest, it's not
easily reproducible and it requires SLUB poisoning to reproduce.
general protection fault: 0000 [#1] SMP
Modules linked in:
CPU: 2 PID: 18421 Comm: userfaultfd Tainted: G ------------ T 3.10.0+ #15
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.1-0-g8891697-prebuilt.qemu-project.org 04/01/2014
task: ffff8801f83b9440 ti: ffff8801f833c000 task.ti: ffff8801f833c000
RIP: 0010:[<ffffffff81451299>] [<ffffffff81451299>] userfaultfd_exit+0x29/0xa0
RSP: 0018:ffff8801f833fe80 EFLAGS: 00010202
RAX: ffff8801f833ffd8 RBX: 6b6b6b6b6b6b6b6b RCX: ffff8801f83b9440
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8800baf18600
RBP: ffff8801f833fee8 R08: 0000000000000000 R09: 0000000000000001
R10: 0000000000000000 R11: ffffffff8127ceb3 R12: 0000000000000000
R13: ffff8800baf186b0 R14: ffff8801f83b99f8 R15: 00007faed746c700
FS: 0000000000000000(0000) GS:ffff88023fc80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007faf0966f028 CR3: 0000000001bc6000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
do_exit+0x297/0xd10
SyS_exit+0x17/0x20
tracesys+0xdd/0xe2
Code: 00 00 66 66 66 66 90 55 48 89 e5 41 54 53 48 83 ec 58 48 8b 1f 48 85 db 75 11 eb 73 66 0f 1f 44 00 00 48 8b 5b 10 48 85 db 74 64 <4c> 8b a3 b8 00 00 00 4d 85 e4 74 eb 41 f6 84 24 2c 01 00 00 80
RIP [<ffffffff81451299>] userfaultfd_exit+0x29/0xa0
RSP <ffff8801f833fe80>
---[ end trace 9fecd6dcb442846a ]---
In the debugger I located the "mm" pointer in the stack and walking
mm->mmap->vm_next through the end shows the vma->vm_next list is fully
consistent and it is null terminated list as expected. So this has to
be an SMP race condition where userfaultfd_exit was running while the
vma list was being modified by another CPU.
When userfaultfd_exit() run one of the ->vm_next pointers pointed to
SLAB_POISON (RBX is the vma pointer and is 0x6b6b..).
The reason is that it's not running in __mmput but while there are still
other threads running and it's not holding the mmap_sem (it can't as it
has to wait the even to be received by the manager). So this is an use
after free that was happening for all processes.
One more implementation problem aside from the race condition:
userfaultfd_exit has really to check a flag in mm->flags before walking
the vma or it's going to slowdown the exit() path for regular tasks.
One more implementation problem: at that point signals can't be
delivered so it would also create a task in D state if the manager
doesn't read the event.
The major design issue: it overall looks superfluous as the manager can
check for -ENOSPC in the background transfer:
if (mmget_not_zero(ctx->mm)) {
[..]
} else {
return -ENOSPC;
}
It's safer to roll it back and re-introduce it later if at all.
[rppt@linux.vnet.ibm.com: documentation fixup after removal of UFFD_EVENT_EXIT]
Link: http://lkml.kernel.org/r/1488345437-4364-1-git-send-email-rppt@linux.vnet.ibm.com
Link: http://lkml.kernel.org/r/20170224181957.19736-2-aarcange@redhat.com
Signed-off-by: Andrea Arcangeli <aarcange@redhat.com>
Signed-off-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Acked-by: Mike Rapoport <rppt@linux.vnet.ibm.com>
Cc: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
Cc: Mike Kravetz <mike.kravetz@oracle.com>
Cc: Pavel Emelyanov <xemul@parallels.com>
Cc: Hillf Danton <hillf.zj@alibaba-inc.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2017-03-10 07:16:49 +07:00
|
|
|
#define UFFD_API_FEATURES (UFFD_FEATURE_EVENT_FORK | \
|
2017-02-23 06:43:19 +07:00
|
|
|
UFFD_FEATURE_EVENT_REMAP | \
|
2017-02-25 05:56:02 +07:00
|
|
|
UFFD_FEATURE_EVENT_REMOVE | \
|
2017-02-25 05:58:22 +07:00
|
|
|
UFFD_FEATURE_EVENT_UNMAP | \
|
2017-02-23 06:43:58 +07:00
|
|
|
UFFD_FEATURE_MISSING_HUGETLBFS | \
|
2017-09-07 06:23:39 +07:00
|
|
|
UFFD_FEATURE_MISSING_SHMEM | \
|
2017-09-07 06:23:56 +07:00
|
|
|
UFFD_FEATURE_SIGBUS | \
|
|
|
|
UFFD_FEATURE_THREAD_ID)
|
2015-09-05 05:46:04 +07:00
|
|
|
#define UFFD_API_IOCTLS \
|
|
|
|
((__u64)1 << _UFFDIO_REGISTER | \
|
|
|
|
(__u64)1 << _UFFDIO_UNREGISTER | \
|
|
|
|
(__u64)1 << _UFFDIO_API)
|
|
|
|
#define UFFD_API_RANGE_IOCTLS \
|
2015-09-05 05:47:01 +07:00
|
|
|
((__u64)1 << _UFFDIO_WAKE | \
|
|
|
|
(__u64)1 << _UFFDIO_COPY | \
|
|
|
|
(__u64)1 << _UFFDIO_ZEROPAGE)
|
2017-02-23 06:43:40 +07:00
|
|
|
#define UFFD_API_RANGE_IOCTLS_BASIC \
|
2017-02-23 06:43:04 +07:00
|
|
|
((__u64)1 << _UFFDIO_WAKE | \
|
|
|
|
(__u64)1 << _UFFDIO_COPY)
|
2015-09-05 05:46:04 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Valid ioctl command number range with this API is from 0x00 to
|
|
|
|
* 0x3F. UFFDIO_API is the fixed number, everything else can be
|
|
|
|
* changed by implementing a different UFFD_API. If sticking to the
|
|
|
|
* same UFFD_API more ioctl can be added and userland will be aware of
|
|
|
|
* which ioctl the running kernel implements through the ioctl command
|
|
|
|
* bitmask written by the UFFDIO_API.
|
|
|
|
*/
|
|
|
|
#define _UFFDIO_REGISTER (0x00)
|
|
|
|
#define _UFFDIO_UNREGISTER (0x01)
|
|
|
|
#define _UFFDIO_WAKE (0x02)
|
2015-09-05 05:47:01 +07:00
|
|
|
#define _UFFDIO_COPY (0x03)
|
|
|
|
#define _UFFDIO_ZEROPAGE (0x04)
|
2015-09-05 05:46:04 +07:00
|
|
|
#define _UFFDIO_API (0x3F)
|
|
|
|
|
|
|
|
/* userfaultfd ioctl ids */
|
|
|
|
#define UFFDIO 0xAA
|
|
|
|
#define UFFDIO_API _IOWR(UFFDIO, _UFFDIO_API, \
|
|
|
|
struct uffdio_api)
|
|
|
|
#define UFFDIO_REGISTER _IOWR(UFFDIO, _UFFDIO_REGISTER, \
|
|
|
|
struct uffdio_register)
|
|
|
|
#define UFFDIO_UNREGISTER _IOR(UFFDIO, _UFFDIO_UNREGISTER, \
|
|
|
|
struct uffdio_range)
|
|
|
|
#define UFFDIO_WAKE _IOR(UFFDIO, _UFFDIO_WAKE, \
|
|
|
|
struct uffdio_range)
|
2015-09-05 05:47:01 +07:00
|
|
|
#define UFFDIO_COPY _IOWR(UFFDIO, _UFFDIO_COPY, \
|
|
|
|
struct uffdio_copy)
|
|
|
|
#define UFFDIO_ZEROPAGE _IOWR(UFFDIO, _UFFDIO_ZEROPAGE, \
|
|
|
|
struct uffdio_zeropage)
|
2015-09-05 05:46:04 +07:00
|
|
|
|
2015-09-05 05:46:37 +07:00
|
|
|
/* read() structure */
|
|
|
|
struct uffd_msg {
|
|
|
|
__u8 event;
|
|
|
|
|
|
|
|
__u8 reserved1;
|
|
|
|
__u16 reserved2;
|
|
|
|
__u32 reserved3;
|
|
|
|
|
|
|
|
union {
|
|
|
|
struct {
|
|
|
|
__u64 flags;
|
|
|
|
__u64 address;
|
2017-09-07 06:23:56 +07:00
|
|
|
__u32 ptid;
|
2015-09-05 05:46:37 +07:00
|
|
|
} pagefault;
|
|
|
|
|
2017-02-23 06:42:27 +07:00
|
|
|
struct {
|
|
|
|
__u32 ufd;
|
|
|
|
} fork;
|
|
|
|
|
2017-02-23 06:42:34 +07:00
|
|
|
struct {
|
|
|
|
__u64 from;
|
|
|
|
__u64 to;
|
|
|
|
__u64 len;
|
|
|
|
} remap;
|
|
|
|
|
2017-02-23 06:42:40 +07:00
|
|
|
struct {
|
|
|
|
__u64 start;
|
|
|
|
__u64 end;
|
2017-02-25 05:56:02 +07:00
|
|
|
} remove;
|
2017-02-23 06:42:40 +07:00
|
|
|
|
2015-09-05 05:46:37 +07:00
|
|
|
struct {
|
|
|
|
/* unused reserved fields */
|
|
|
|
__u64 reserved1;
|
|
|
|
__u64 reserved2;
|
|
|
|
__u64 reserved3;
|
|
|
|
} reserved;
|
|
|
|
} arg;
|
|
|
|
} __packed;
|
2015-09-05 05:46:04 +07:00
|
|
|
|
2015-09-05 05:46:34 +07:00
|
|
|
/*
|
2015-09-05 05:46:37 +07:00
|
|
|
* Start at 0x12 and not at 0 to be more strict against bugs.
|
2015-09-05 05:46:34 +07:00
|
|
|
*/
|
2015-09-05 05:46:37 +07:00
|
|
|
#define UFFD_EVENT_PAGEFAULT 0x12
|
|
|
|
#define UFFD_EVENT_FORK 0x13
|
2017-02-23 06:42:34 +07:00
|
|
|
#define UFFD_EVENT_REMAP 0x14
|
2017-02-25 05:56:02 +07:00
|
|
|
#define UFFD_EVENT_REMOVE 0x15
|
2017-02-25 05:58:22 +07:00
|
|
|
#define UFFD_EVENT_UNMAP 0x16
|
2015-09-05 05:46:37 +07:00
|
|
|
|
|
|
|
/* flags for UFFD_EVENT_PAGEFAULT */
|
|
|
|
#define UFFD_PAGEFAULT_FLAG_WRITE (1<<0) /* If this was a write fault */
|
|
|
|
#define UFFD_PAGEFAULT_FLAG_WP (1<<1) /* If reason is VM_UFFD_WP */
|
2015-09-05 05:46:34 +07:00
|
|
|
|
2015-09-05 05:46:04 +07:00
|
|
|
struct uffdio_api {
|
2015-09-05 05:46:37 +07:00
|
|
|
/* userland asks for an API number and the features to enable */
|
2015-09-05 05:46:04 +07:00
|
|
|
__u64 api;
|
2015-09-05 05:46:37 +07:00
|
|
|
/*
|
|
|
|
* Kernel answers below with the all available features for
|
|
|
|
* the API, this notifies userland of which events and/or
|
|
|
|
* which flags for each event are enabled in the current
|
|
|
|
* kernel.
|
|
|
|
*
|
|
|
|
* Note: UFFD_EVENT_PAGEFAULT and UFFD_PAGEFAULT_FLAG_WRITE
|
|
|
|
* are to be considered implicitly always enabled in all kernels as
|
|
|
|
* long as the uffdio_api.api requested matches UFFD_API.
|
2017-02-23 06:43:19 +07:00
|
|
|
*
|
|
|
|
* UFFD_FEATURE_MISSING_HUGETLBFS means an UFFDIO_REGISTER
|
|
|
|
* with UFFDIO_REGISTER_MODE_MISSING mode will succeed on
|
|
|
|
* hugetlbfs virtual memory ranges. Adding or not adding
|
|
|
|
* UFFD_FEATURE_MISSING_HUGETLBFS to uffdio_api.features has
|
|
|
|
* no real functional effect after UFFDIO_API returns, but
|
|
|
|
* it's only useful for an initial feature set probe at
|
|
|
|
* UFFDIO_API time. There are two ways to use it:
|
|
|
|
*
|
|
|
|
* 1) by adding UFFD_FEATURE_MISSING_HUGETLBFS to the
|
|
|
|
* uffdio_api.features before calling UFFDIO_API, an error
|
|
|
|
* will be returned by UFFDIO_API on a kernel without
|
|
|
|
* hugetlbfs missing support
|
|
|
|
*
|
|
|
|
* 2) the UFFD_FEATURE_MISSING_HUGETLBFS can not be added in
|
|
|
|
* uffdio_api.features and instead it will be set by the
|
|
|
|
* kernel in the uffdio_api.features if the kernel supports
|
|
|
|
* it, so userland can later check if the feature flag is
|
|
|
|
* present in uffdio_api.features after UFFDIO_API
|
|
|
|
* succeeded.
|
2017-02-23 06:43:58 +07:00
|
|
|
*
|
|
|
|
* UFFD_FEATURE_MISSING_SHMEM works the same as
|
|
|
|
* UFFD_FEATURE_MISSING_HUGETLBFS, but it applies to shmem
|
|
|
|
* (i.e. tmpfs and other shmem based APIs).
|
2017-09-07 06:23:39 +07:00
|
|
|
*
|
|
|
|
* UFFD_FEATURE_SIGBUS feature means no page-fault
|
|
|
|
* (UFFD_EVENT_PAGEFAULT) event will be delivered, instead
|
|
|
|
* a SIGBUS signal will be sent to the faulting process.
|
2017-09-07 06:23:56 +07:00
|
|
|
*
|
|
|
|
* UFFD_FEATURE_THREAD_ID pid of the page faulted task_struct will
|
|
|
|
* be returned, if feature is not requested 0 will be returned.
|
2015-09-05 05:46:37 +07:00
|
|
|
*/
|
|
|
|
#define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0)
|
|
|
|
#define UFFD_FEATURE_EVENT_FORK (1<<1)
|
2017-02-23 06:42:34 +07:00
|
|
|
#define UFFD_FEATURE_EVENT_REMAP (1<<2)
|
2017-02-25 05:56:02 +07:00
|
|
|
#define UFFD_FEATURE_EVENT_REMOVE (1<<3)
|
2017-02-23 06:43:19 +07:00
|
|
|
#define UFFD_FEATURE_MISSING_HUGETLBFS (1<<4)
|
2017-02-23 06:43:58 +07:00
|
|
|
#define UFFD_FEATURE_MISSING_SHMEM (1<<5)
|
2017-02-25 05:58:22 +07:00
|
|
|
#define UFFD_FEATURE_EVENT_UNMAP (1<<6)
|
2017-09-07 06:23:39 +07:00
|
|
|
#define UFFD_FEATURE_SIGBUS (1<<7)
|
2017-09-07 06:23:56 +07:00
|
|
|
#define UFFD_FEATURE_THREAD_ID (1<<8)
|
2015-09-05 05:46:34 +07:00
|
|
|
__u64 features;
|
2015-09-05 05:46:37 +07:00
|
|
|
|
2015-09-05 05:46:04 +07:00
|
|
|
__u64 ioctls;
|
|
|
|
};
|
|
|
|
|
|
|
|
struct uffdio_range {
|
|
|
|
__u64 start;
|
|
|
|
__u64 len;
|
|
|
|
};
|
|
|
|
|
|
|
|
struct uffdio_register {
|
|
|
|
struct uffdio_range range;
|
|
|
|
#define UFFDIO_REGISTER_MODE_MISSING ((__u64)1<<0)
|
|
|
|
#define UFFDIO_REGISTER_MODE_WP ((__u64)1<<1)
|
|
|
|
__u64 mode;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* kernel answers which ioctl commands are available for the
|
|
|
|
* range, keep at the end as the last 8 bytes aren't read.
|
|
|
|
*/
|
|
|
|
__u64 ioctls;
|
|
|
|
};
|
|
|
|
|
2015-09-05 05:47:01 +07:00
|
|
|
struct uffdio_copy {
|
|
|
|
__u64 dst;
|
|
|
|
__u64 src;
|
|
|
|
__u64 len;
|
|
|
|
/*
|
|
|
|
* There will be a wrprotection flag later that allows to map
|
|
|
|
* pages wrprotected on the fly. And such a flag will be
|
|
|
|
* available if the wrprotection ioctl are implemented for the
|
|
|
|
* range according to the uffdio_register.ioctls.
|
|
|
|
*/
|
|
|
|
#define UFFDIO_COPY_MODE_DONTWAKE ((__u64)1<<0)
|
|
|
|
__u64 mode;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* "copy" is written by the ioctl and must be at the end: the
|
|
|
|
* copy_from_user will not read the last 8 bytes.
|
|
|
|
*/
|
|
|
|
__s64 copy;
|
|
|
|
};
|
|
|
|
|
|
|
|
struct uffdio_zeropage {
|
|
|
|
struct uffdio_range range;
|
|
|
|
#define UFFDIO_ZEROPAGE_MODE_DONTWAKE ((__u64)1<<0)
|
|
|
|
__u64 mode;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* "zeropage" is written by the ioctl and must be at the end:
|
|
|
|
* the copy_from_user will not read the last 8 bytes.
|
|
|
|
*/
|
|
|
|
__s64 zeropage;
|
|
|
|
};
|
|
|
|
|
2015-09-05 05:46:04 +07:00
|
|
|
#endif /* _LINUX_USERFAULTFD_H */
|