2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* linux/fs/namei.c
|
|
|
|
*
|
|
|
|
* Copyright (C) 1991, 1992 Linus Torvalds
|
|
|
|
*/
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Some corrections by tytso.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* [Feb 1997 T. Schoebel-Theuer] Complete rewrite of the pathname
|
|
|
|
* lookup logic.
|
|
|
|
*/
|
|
|
|
/* [Feb-Apr 2000, AV] Rewrite to the new namespace architecture.
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <linux/init.h>
|
2011-11-17 11:57:37 +07:00
|
|
|
#include <linux/export.h>
|
2012-05-24 10:12:50 +07:00
|
|
|
#include <linux/kernel.h>
|
2005-04-17 05:20:36 +07:00
|
|
|
#include <linux/slab.h>
|
|
|
|
#include <linux/fs.h>
|
|
|
|
#include <linux/namei.h>
|
|
|
|
#include <linux/pagemap.h>
|
[PATCH] inotify
inotify is intended to correct the deficiencies of dnotify, particularly
its inability to scale and its terrible user interface:
* dnotify requires the opening of one fd per each directory
that you intend to watch. This quickly results in too many
open files and pins removable media, preventing unmount.
* dnotify is directory-based. You only learn about changes to
directories. Sure, a change to a file in a directory affects
the directory, but you are then forced to keep a cache of
stat structures.
* dnotify's interface to user-space is awful. Signals?
inotify provides a more usable, simple, powerful solution to file change
notification:
* inotify's interface is a system call that returns a fd, not SIGIO.
You get a single fd, which is select()-able.
* inotify has an event that says "the filesystem that the item
you were watching is on was unmounted."
* inotify can watch directories or files.
Inotify is currently used by Beagle (a desktop search infrastructure),
Gamin (a FAM replacement), and other projects.
See Documentation/filesystems/inotify.txt.
Signed-off-by: Robert Love <rml@novell.com>
Cc: John McCutchan <ttb@tentacle.dhs.org>
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-07-13 04:06:03 +07:00
|
|
|
#include <linux/fsnotify.h>
|
2005-04-17 05:20:36 +07:00
|
|
|
#include <linux/personality.h>
|
|
|
|
#include <linux/security.h>
|
2009-02-04 21:06:57 +07:00
|
|
|
#include <linux/ima.h>
|
2005-04-17 05:20:36 +07:00
|
|
|
#include <linux/syscalls.h>
|
|
|
|
#include <linux/mount.h>
|
|
|
|
#include <linux/audit.h>
|
2006-01-12 03:17:46 +07:00
|
|
|
#include <linux/capability.h>
|
2005-10-19 04:20:16 +07:00
|
|
|
#include <linux/file.h>
|
2006-01-19 08:43:53 +07:00
|
|
|
#include <linux/fcntl.h>
|
2008-04-29 15:00:10 +07:00
|
|
|
#include <linux/device_cgroup.h>
|
2009-03-30 06:50:06 +07:00
|
|
|
#include <linux/fs_struct.h>
|
2011-07-23 09:30:19 +07:00
|
|
|
#include <linux/posix_acl.h>
|
2005-04-17 05:20:36 +07:00
|
|
|
#include <asm/uaccess.h>
|
|
|
|
|
2009-12-05 03:47:36 +07:00
|
|
|
#include "internal.h"
|
2011-11-25 06:22:03 +07:00
|
|
|
#include "mount.h"
|
2009-12-05 03:47:36 +07:00
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
/* [Feb-1997 T. Schoebel-Theuer]
|
|
|
|
* Fundamental changes in the pathname lookup mechanisms (namei)
|
|
|
|
* were necessary because of omirr. The reason is that omirr needs
|
|
|
|
* to know the _real_ pathname, not the user-supplied one, in case
|
|
|
|
* of symlinks (and also when transname replacements occur).
|
|
|
|
*
|
|
|
|
* The new code replaces the old recursive symlink resolution with
|
|
|
|
* an iterative one (in case of non-nested symlink chains). It does
|
|
|
|
* this with calls to <fs>_follow_link().
|
|
|
|
* As a side effect, dir_namei(), _namei() and follow_link() are now
|
|
|
|
* replaced with a single function lookup_dentry() that can handle all
|
|
|
|
* the special cases of the former code.
|
|
|
|
*
|
|
|
|
* With the new dcache, the pathname is stored at each inode, at least as
|
|
|
|
* long as the refcount of the inode is positive. As a side effect, the
|
|
|
|
* size of the dcache depends on the inode cache and thus is dynamic.
|
|
|
|
*
|
|
|
|
* [29-Apr-1998 C. Scott Ananian] Updated above description of symlink
|
|
|
|
* resolution to correspond with current state of the code.
|
|
|
|
*
|
|
|
|
* Note that the symlink resolution is not *completely* iterative.
|
|
|
|
* There is still a significant amount of tail- and mid- recursion in
|
|
|
|
* the algorithm. Also, note that <fs>_readlink() is not used in
|
|
|
|
* lookup_dentry(): lookup_dentry() on the result of <fs>_readlink()
|
|
|
|
* may return different results than <fs>_follow_link(). Many virtual
|
|
|
|
* filesystems (including /proc) exhibit this behavior.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* [24-Feb-97 T. Schoebel-Theuer] Side effects caused by new implementation:
|
|
|
|
* New symlink semantics: when open() is called with flags O_CREAT | O_EXCL
|
|
|
|
* and the name already exists in form of a symlink, try to create the new
|
|
|
|
* name indicated by the symlink. The old code always complained that the
|
|
|
|
* name already exists, due to not following the symlink even if its target
|
|
|
|
* is nonexistent. The new semantics affects also mknod() and link() when
|
2011-03-31 08:57:33 +07:00
|
|
|
* the name is a symlink pointing to a non-existent name.
|
2005-04-17 05:20:36 +07:00
|
|
|
*
|
|
|
|
* I don't know which semantics is the right one, since I have no access
|
|
|
|
* to standards. But I found by trial that HP-UX 9.0 has the full "new"
|
|
|
|
* semantics implemented, while SunOS 4.1.1 and Solaris (SunOS 5.4) have the
|
|
|
|
* "old" one. Personally, I think the new semantics is much more logical.
|
|
|
|
* Note that "ln old new" where "new" is a symlink pointing to a non-existing
|
|
|
|
* file does succeed in both HP-UX and SunOs, but not in Solaris
|
|
|
|
* and in the old Linux semantics.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* [16-Dec-97 Kevin Buhr] For security reasons, we change some symlink
|
|
|
|
* semantics. See the comments in "open_namei" and "do_link" below.
|
|
|
|
*
|
|
|
|
* [10-Sep-98 Alan Modra] Another symlink change.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* [Feb-Apr 2000 AV] Complete rewrite. Rules for symlinks:
|
|
|
|
* inside the path - always follow.
|
|
|
|
* in the last component in creation/removal/renaming - never follow.
|
|
|
|
* if LOOKUP_FOLLOW passed - follow.
|
|
|
|
* if the pathname has trailing slashes - follow.
|
|
|
|
* otherwise - don't follow.
|
|
|
|
* (applied in that order).
|
|
|
|
*
|
|
|
|
* [Jun 2000 AV] Inconsistent behaviour of open() in case if flags==O_CREAT
|
|
|
|
* restored for 2.4. This is the last surviving part of old 4.2BSD bug.
|
|
|
|
* During the 2.4 we need to fix the userland stuff depending on it -
|
|
|
|
* hopefully we will be able to get rid of that wart in 2.5. So far only
|
|
|
|
* XEmacs seems to be relying on it...
|
|
|
|
*/
|
|
|
|
/*
|
|
|
|
* [Sep 2001 AV] Single-semaphore locking scheme (kudos to David Holland)
|
2006-03-23 18:00:33 +07:00
|
|
|
* implemented. Let's see if raised priority of ->s_vfs_rename_mutex gives
|
2005-04-17 05:20:36 +07:00
|
|
|
* any extra contention...
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* In order to reduce some races, while at the same time doing additional
|
|
|
|
* checking and hopefully speeding things up, we copy filenames to the
|
|
|
|
* kernel data space before using them..
|
|
|
|
*
|
|
|
|
* POSIX.1 2.4: an empty pathname is invalid (ENOENT).
|
|
|
|
* PATH_MAX includes the nul terminator --RR.
|
|
|
|
*/
|
2011-11-02 15:44:39 +07:00
|
|
|
static char *getname_flags(const char __user *filename, int flags, int *empty)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2012-04-29 04:38:32 +07:00
|
|
|
char *result = __getname(), *err;
|
|
|
|
int len;
|
2012-01-04 02:23:08 +07:00
|
|
|
|
2012-04-29 04:38:32 +07:00
|
|
|
if (unlikely(!result))
|
2012-01-04 02:23:08 +07:00
|
|
|
return ERR_PTR(-ENOMEM);
|
|
|
|
|
2012-04-29 04:38:32 +07:00
|
|
|
len = strncpy_from_user(result, filename, PATH_MAX);
|
|
|
|
err = ERR_PTR(len);
|
|
|
|
if (unlikely(len < 0))
|
|
|
|
goto error;
|
|
|
|
|
|
|
|
/* The empty path is special. */
|
|
|
|
if (unlikely(!len)) {
|
|
|
|
if (empty)
|
2012-01-04 02:23:08 +07:00
|
|
|
*empty = 1;
|
2012-04-29 04:38:32 +07:00
|
|
|
err = ERR_PTR(-ENOENT);
|
|
|
|
if (!(flags & LOOKUP_EMPTY))
|
|
|
|
goto error;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
2012-04-29 04:38:32 +07:00
|
|
|
|
|
|
|
err = ERR_PTR(-ENAMETOOLONG);
|
|
|
|
if (likely(len < PATH_MAX)) {
|
|
|
|
audit_getname(result);
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
|
|
|
error:
|
|
|
|
__putname(result);
|
|
|
|
return err;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2011-03-15 05:56:51 +07:00
|
|
|
char *getname(const char __user * filename)
|
|
|
|
{
|
2012-03-23 06:10:40 +07:00
|
|
|
return getname_flags(filename, 0, NULL);
|
2011-03-15 05:56:51 +07:00
|
|
|
}
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
#ifdef CONFIG_AUDITSYSCALL
|
|
|
|
void putname(const char *name)
|
|
|
|
{
|
2006-07-16 17:38:45 +07:00
|
|
|
if (unlikely(!audit_dummy_context()))
|
2005-04-17 05:20:36 +07:00
|
|
|
audit_putname(name);
|
|
|
|
else
|
|
|
|
__putname(name);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(putname);
|
|
|
|
#endif
|
|
|
|
|
2011-07-23 09:30:19 +07:00
|
|
|
static int check_acl(struct inode *inode, int mask)
|
|
|
|
{
|
2011-07-26 12:47:03 +07:00
|
|
|
#ifdef CONFIG_FS_POSIX_ACL
|
2011-07-23 09:30:19 +07:00
|
|
|
struct posix_acl *acl;
|
|
|
|
|
|
|
|
if (mask & MAY_NOT_BLOCK) {
|
2011-08-03 08:32:13 +07:00
|
|
|
acl = get_cached_acl_rcu(inode, ACL_TYPE_ACCESS);
|
|
|
|
if (!acl)
|
2011-07-23 09:30:19 +07:00
|
|
|
return -EAGAIN;
|
2011-08-03 08:32:13 +07:00
|
|
|
/* no ->get_acl() calls in RCU mode... */
|
|
|
|
if (acl == ACL_NOT_CACHED)
|
|
|
|
return -ECHILD;
|
2011-08-06 23:43:07 +07:00
|
|
|
return posix_acl_permission(inode, acl, mask & ~MAY_NOT_BLOCK);
|
2011-07-23 09:30:19 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
acl = get_cached_acl(inode, ACL_TYPE_ACCESS);
|
|
|
|
|
|
|
|
/*
|
2011-07-23 22:37:31 +07:00
|
|
|
* A filesystem can force a ACL callback by just never filling the
|
|
|
|
* ACL cache. But normally you'd fill the cache either at inode
|
|
|
|
* instantiation time, or on the first ->get_acl call.
|
2011-07-23 09:30:19 +07:00
|
|
|
*
|
2011-07-23 22:37:31 +07:00
|
|
|
* If the filesystem doesn't have a get_acl() function at all, we'll
|
|
|
|
* just create the negative cache entry.
|
2011-07-23 09:30:19 +07:00
|
|
|
*/
|
|
|
|
if (acl == ACL_NOT_CACHED) {
|
2011-07-23 22:37:31 +07:00
|
|
|
if (inode->i_op->get_acl) {
|
|
|
|
acl = inode->i_op->get_acl(inode, ACL_TYPE_ACCESS);
|
|
|
|
if (IS_ERR(acl))
|
|
|
|
return PTR_ERR(acl);
|
|
|
|
} else {
|
|
|
|
set_cached_acl(inode, ACL_TYPE_ACCESS, NULL);
|
|
|
|
return -EAGAIN;
|
|
|
|
}
|
2011-07-23 09:30:19 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
if (acl) {
|
|
|
|
int error = posix_acl_permission(inode, acl, mask);
|
|
|
|
posix_acl_release(acl);
|
|
|
|
return error;
|
|
|
|
}
|
2011-07-26 12:47:03 +07:00
|
|
|
#endif
|
2011-07-23 09:30:19 +07:00
|
|
|
|
|
|
|
return -EAGAIN;
|
|
|
|
}
|
|
|
|
|
2009-08-29 01:51:25 +07:00
|
|
|
/*
|
2011-10-24 00:43:33 +07:00
|
|
|
* This does the basic permission checking
|
2005-04-17 05:20:36 +07:00
|
|
|
*/
|
2011-06-21 06:12:17 +07:00
|
|
|
static int acl_permission_check(struct inode *inode, int mask)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2011-05-14 01:51:01 +07:00
|
|
|
unsigned int mode = inode->i_mode;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2012-03-04 12:17:15 +07:00
|
|
|
if (likely(uid_eq(current_fsuid(), inode->i_uid)))
|
2005-04-17 05:20:36 +07:00
|
|
|
mode >>= 6;
|
|
|
|
else {
|
2011-07-23 09:30:19 +07:00
|
|
|
if (IS_POSIXACL(inode) && (mode & S_IRWXG)) {
|
2011-06-21 06:12:17 +07:00
|
|
|
int error = check_acl(inode, mask);
|
2011-01-07 13:49:58 +07:00
|
|
|
if (error != -EAGAIN)
|
|
|
|
return error;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
if (in_group_p(inode->i_gid))
|
|
|
|
mode >>= 3;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If the DACs are ok we don't need any capability check.
|
|
|
|
*/
|
2011-06-21 06:06:22 +07:00
|
|
|
if ((mask & ~mode & (MAY_READ | MAY_WRITE | MAY_EXEC)) == 0)
|
2005-04-17 05:20:36 +07:00
|
|
|
return 0;
|
2009-08-29 01:51:25 +07:00
|
|
|
return -EACCES;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
2011-01-07 13:49:58 +07:00
|
|
|
* generic_permission - check for access rights on a Posix-like filesystem
|
2009-08-29 01:51:25 +07:00
|
|
|
* @inode: inode to check access rights for
|
2011-10-24 00:43:30 +07:00
|
|
|
* @mask: right to check for (%MAY_READ, %MAY_WRITE, %MAY_EXEC, ...)
|
2009-08-29 01:51:25 +07:00
|
|
|
*
|
|
|
|
* Used to check for read/write/execute permissions on a file.
|
|
|
|
* We use "fsuid" for this, letting us set arbitrary permissions
|
|
|
|
* for filesystem access without changing the "normal" uids which
|
2011-01-07 13:49:58 +07:00
|
|
|
* are used for other things.
|
|
|
|
*
|
|
|
|
* generic_permission is rcu-walk aware. It returns -ECHILD in case an rcu-walk
|
|
|
|
* request cannot be satisfied (eg. requires blocking or too much complexity).
|
|
|
|
* It would then be called again in ref-walk mode.
|
2009-08-29 01:51:25 +07:00
|
|
|
*/
|
2011-06-21 06:16:29 +07:00
|
|
|
int generic_permission(struct inode *inode, int mask)
|
2009-08-29 01:51:25 +07:00
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
/*
|
2011-10-24 00:43:33 +07:00
|
|
|
* Do the basic permission checks.
|
2009-08-29 01:51:25 +07:00
|
|
|
*/
|
2011-06-21 06:12:17 +07:00
|
|
|
ret = acl_permission_check(inode, mask);
|
2009-08-29 01:51:25 +07:00
|
|
|
if (ret != -EACCES)
|
|
|
|
return ret;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2011-06-21 06:55:42 +07:00
|
|
|
if (S_ISDIR(inode->i_mode)) {
|
|
|
|
/* DACs are overridable for directories */
|
2011-11-15 07:24:06 +07:00
|
|
|
if (inode_capable(inode, CAP_DAC_OVERRIDE))
|
2011-06-21 06:55:42 +07:00
|
|
|
return 0;
|
|
|
|
if (!(mask & MAY_WRITE))
|
2011-11-15 07:24:06 +07:00
|
|
|
if (inode_capable(inode, CAP_DAC_READ_SEARCH))
|
2011-06-21 06:55:42 +07:00
|
|
|
return 0;
|
|
|
|
return -EACCES;
|
|
|
|
}
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* Read/write DACs are always overridable.
|
2011-06-21 06:55:42 +07:00
|
|
|
* Executable DACs are overridable when there is
|
|
|
|
* at least one exec bit set.
|
2005-04-17 05:20:36 +07:00
|
|
|
*/
|
2011-06-21 06:55:42 +07:00
|
|
|
if (!(mask & MAY_EXEC) || (inode->i_mode & S_IXUGO))
|
2011-11-15 07:24:06 +07:00
|
|
|
if (inode_capable(inode, CAP_DAC_OVERRIDE))
|
2005-04-17 05:20:36 +07:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Searching includes executable on directories, else just read.
|
|
|
|
*/
|
2009-12-30 03:50:19 +07:00
|
|
|
mask &= MAY_READ | MAY_WRITE | MAY_EXEC;
|
2011-06-21 06:55:42 +07:00
|
|
|
if (mask == MAY_READ)
|
2011-11-15 07:24:06 +07:00
|
|
|
if (inode_capable(inode, CAP_DAC_READ_SEARCH))
|
2005-04-17 05:20:36 +07:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
return -EACCES;
|
|
|
|
}
|
|
|
|
|
2011-08-07 12:45:50 +07:00
|
|
|
/*
|
|
|
|
* We _really_ want to just do "generic_permission()" without
|
|
|
|
* even looking at the inode->i_op values. So we keep a cache
|
|
|
|
* flag in inode->i_opflags, that says "this has not special
|
|
|
|
* permission function, use the fast case".
|
|
|
|
*/
|
|
|
|
static inline int do_inode_permission(struct inode *inode, int mask)
|
|
|
|
{
|
|
|
|
if (unlikely(!(inode->i_opflags & IOP_FASTPERM))) {
|
|
|
|
if (likely(inode->i_op->permission))
|
|
|
|
return inode->i_op->permission(inode, mask);
|
|
|
|
|
|
|
|
/* This gets set once for the inode lifetime */
|
|
|
|
spin_lock(&inode->i_lock);
|
|
|
|
inode->i_opflags |= IOP_FASTPERM;
|
|
|
|
spin_unlock(&inode->i_lock);
|
|
|
|
}
|
|
|
|
return generic_permission(inode, mask);
|
|
|
|
}
|
|
|
|
|
2008-10-24 14:59:29 +07:00
|
|
|
/**
|
|
|
|
* inode_permission - check for access rights to a given inode
|
|
|
|
* @inode: inode to check permission on
|
2011-10-24 00:43:30 +07:00
|
|
|
* @mask: right to check for (%MAY_READ, %MAY_WRITE, %MAY_EXEC, ...)
|
2008-10-24 14:59:29 +07:00
|
|
|
*
|
|
|
|
* Used to check for read/write/execute permissions on an inode.
|
|
|
|
* We use "fsuid" for this, letting us set arbitrary permissions
|
|
|
|
* for filesystem access without changing the "normal" uids which
|
|
|
|
* are used for other things.
|
2011-10-24 00:43:33 +07:00
|
|
|
*
|
|
|
|
* When checking for MAY_APPEND, MAY_WRITE must also be set in @mask.
|
2008-10-24 14:59:29 +07:00
|
|
|
*/
|
2008-07-22 11:07:17 +07:00
|
|
|
int inode_permission(struct inode *inode, int mask)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2008-07-16 08:03:57 +07:00
|
|
|
int retval;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2011-08-07 12:45:50 +07:00
|
|
|
if (unlikely(mask & MAY_WRITE)) {
|
2007-10-17 13:27:08 +07:00
|
|
|
umode_t mode = inode->i_mode;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Nobody gets write access to a read-only fs.
|
|
|
|
*/
|
|
|
|
if (IS_RDONLY(inode) &&
|
|
|
|
(S_ISREG(mode) || S_ISDIR(mode) || S_ISLNK(mode)))
|
|
|
|
return -EROFS;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Nobody gets write access to an immutable file.
|
|
|
|
*/
|
|
|
|
if (IS_IMMUTABLE(inode))
|
|
|
|
return -EACCES;
|
|
|
|
}
|
|
|
|
|
2011-08-07 12:45:50 +07:00
|
|
|
retval = do_inode_permission(inode, mask);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (retval)
|
|
|
|
return retval;
|
|
|
|
|
2008-04-29 15:00:10 +07:00
|
|
|
retval = devcgroup_inode_permission(inode, mask);
|
|
|
|
if (retval)
|
|
|
|
return retval;
|
|
|
|
|
2010-07-23 22:43:57 +07:00
|
|
|
return security_inode_permission(inode, mask);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2008-02-15 10:34:38 +07:00
|
|
|
/**
|
|
|
|
* path_get - get a reference to a path
|
|
|
|
* @path: path to get the reference to
|
|
|
|
*
|
|
|
|
* Given a path increment the reference count to the dentry and the vfsmount.
|
|
|
|
*/
|
|
|
|
void path_get(struct path *path)
|
|
|
|
{
|
|
|
|
mntget(path->mnt);
|
|
|
|
dget(path->dentry);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(path_get);
|
|
|
|
|
2008-02-15 10:34:35 +07:00
|
|
|
/**
|
|
|
|
* path_put - put a reference to a path
|
|
|
|
* @path: path to put the reference to
|
|
|
|
*
|
|
|
|
* Given a path decrement the reference count to the dentry and the vfsmount.
|
|
|
|
*/
|
|
|
|
void path_put(struct path *path)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2008-02-15 10:34:35 +07:00
|
|
|
dput(path->dentry);
|
|
|
|
mntput(path->mnt);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
2008-02-15 10:34:35 +07:00
|
|
|
EXPORT_SYMBOL(path_put);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2011-03-25 21:32:48 +07:00
|
|
|
/*
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
* Path walking has 2 modes, rcu-walk and ref-walk (see
|
2011-03-25 21:32:48 +07:00
|
|
|
* Documentation/filesystems/path-lookup.txt). In situations when we can't
|
|
|
|
* continue in RCU mode, we attempt to drop out of rcu-walk mode and grab
|
|
|
|
* normal reference counts on dentries and vfsmounts to transition to rcu-walk
|
|
|
|
* mode. Refcounts are grabbed at the last known good point before rcu-walk
|
|
|
|
* got stuck, so ref-walk may continue from there. If this is not successful
|
|
|
|
* (eg. a seqcount has changed), then failure is returned and it's up to caller
|
|
|
|
* to restart the path walk from the beginning in ref-walk mode.
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
*/
|
|
|
|
|
|
|
|
/**
|
2011-03-25 21:32:48 +07:00
|
|
|
* unlazy_walk - try to switch to ref-walk mode.
|
|
|
|
* @nd: nameidata pathwalk data
|
|
|
|
* @dentry: child of nd->path.dentry or NULL
|
2011-01-09 10:36:21 +07:00
|
|
|
* Returns: 0 on success, -ECHILD on failure
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
*
|
2011-03-25 21:32:48 +07:00
|
|
|
* unlazy_walk attempts to legitimize the current nd->path, nd->root and dentry
|
|
|
|
* for ref-walk mode. @dentry must be a path found by a do_lookup call on
|
|
|
|
* @nd or NULL. Must be called from rcu-walk context.
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
*/
|
2011-03-25 21:32:48 +07:00
|
|
|
static int unlazy_walk(struct nameidata *nd, struct dentry *dentry)
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
{
|
|
|
|
struct fs_struct *fs = current->fs;
|
|
|
|
struct dentry *parent = nd->path.dentry;
|
2011-03-10 11:04:47 +07:00
|
|
|
int want_root = 0;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
|
|
|
|
BUG_ON(!(nd->flags & LOOKUP_RCU));
|
2011-03-10 11:04:47 +07:00
|
|
|
if (nd->root.mnt && !(nd->flags & LOOKUP_ROOT)) {
|
|
|
|
want_root = 1;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
spin_lock(&fs->lock);
|
|
|
|
if (nd->root.mnt != fs->root.mnt ||
|
|
|
|
nd->root.dentry != fs->root.dentry)
|
|
|
|
goto err_root;
|
|
|
|
}
|
|
|
|
spin_lock(&parent->d_lock);
|
2011-03-25 21:32:48 +07:00
|
|
|
if (!dentry) {
|
|
|
|
if (!__d_rcu_to_refcount(parent, nd->seq))
|
|
|
|
goto err_parent;
|
|
|
|
BUG_ON(nd->inode != parent->d_inode);
|
|
|
|
} else {
|
2011-07-13 08:40:23 +07:00
|
|
|
if (dentry->d_parent != parent)
|
|
|
|
goto err_parent;
|
2011-03-25 21:32:48 +07:00
|
|
|
spin_lock_nested(&dentry->d_lock, DENTRY_D_LOCK_NESTED);
|
|
|
|
if (!__d_rcu_to_refcount(dentry, nd->seq))
|
|
|
|
goto err_child;
|
|
|
|
/*
|
|
|
|
* If the sequence check on the child dentry passed, then
|
|
|
|
* the child has not been removed from its parent. This
|
|
|
|
* means the parent dentry must be valid and able to take
|
|
|
|
* a reference at this point.
|
|
|
|
*/
|
|
|
|
BUG_ON(!IS_ROOT(dentry) && dentry->d_parent != parent);
|
|
|
|
BUG_ON(!parent->d_count);
|
|
|
|
parent->d_count++;
|
|
|
|
spin_unlock(&dentry->d_lock);
|
|
|
|
}
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
spin_unlock(&parent->d_lock);
|
2011-03-10 11:04:47 +07:00
|
|
|
if (want_root) {
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
path_get(&nd->root);
|
|
|
|
spin_unlock(&fs->lock);
|
|
|
|
}
|
|
|
|
mntget(nd->path.mnt);
|
|
|
|
|
|
|
|
rcu_read_unlock();
|
2012-05-08 11:02:02 +07:00
|
|
|
br_read_unlock(&vfsmount_lock);
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
nd->flags &= ~LOOKUP_RCU;
|
|
|
|
return 0;
|
2011-03-25 21:32:48 +07:00
|
|
|
|
|
|
|
err_child:
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
spin_unlock(&dentry->d_lock);
|
2011-03-25 21:32:48 +07:00
|
|
|
err_parent:
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
spin_unlock(&parent->d_lock);
|
|
|
|
err_root:
|
2011-03-10 11:04:47 +07:00
|
|
|
if (want_root)
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
spin_unlock(&fs->lock);
|
|
|
|
return -ECHILD;
|
|
|
|
}
|
|
|
|
|
2005-10-19 04:20:16 +07:00
|
|
|
/**
|
|
|
|
* release_open_intent - free up open intent resources
|
|
|
|
* @nd: pointer to nameidata
|
|
|
|
*/
|
|
|
|
void release_open_intent(struct nameidata *nd)
|
|
|
|
{
|
2011-02-12 06:53:38 +07:00
|
|
|
struct file *file = nd->intent.open.file;
|
|
|
|
|
|
|
|
if (file && !IS_ERR(file)) {
|
|
|
|
if (file->f_path.dentry == NULL)
|
|
|
|
put_filp(file);
|
|
|
|
else
|
|
|
|
fput(file);
|
|
|
|
}
|
2005-10-19 04:20:16 +07:00
|
|
|
}
|
|
|
|
|
2011-02-15 13:35:28 +07:00
|
|
|
static inline int d_revalidate(struct dentry *dentry, struct nameidata *nd)
|
2011-01-07 13:49:57 +07:00
|
|
|
{
|
2011-02-15 13:35:28 +07:00
|
|
|
return dentry->d_op->d_revalidate(dentry, nd);
|
2011-01-07 13:49:57 +07:00
|
|
|
}
|
|
|
|
|
2011-03-25 22:00:12 +07:00
|
|
|
/**
|
|
|
|
* complete_walk - successful completion of path walk
|
|
|
|
* @nd: pointer nameidata
|
2009-12-08 00:01:50 +07:00
|
|
|
*
|
2011-03-25 22:00:12 +07:00
|
|
|
* If we had been in RCU mode, drop out of it and legitimize nd->path.
|
|
|
|
* Revalidate the final result, unless we'd already done that during
|
|
|
|
* the path walk or the filesystem doesn't ask for it. Return 0 on
|
|
|
|
* success, -error on failure. In case of failure caller does not
|
|
|
|
* need to drop nd->path.
|
2009-12-08 00:01:50 +07:00
|
|
|
*/
|
2011-03-25 22:00:12 +07:00
|
|
|
static int complete_walk(struct nameidata *nd)
|
2009-12-08 00:01:50 +07:00
|
|
|
{
|
2011-02-23 03:50:10 +07:00
|
|
|
struct dentry *dentry = nd->path.dentry;
|
2009-12-08 00:01:50 +07:00
|
|
|
int status;
|
|
|
|
|
2011-03-25 22:00:12 +07:00
|
|
|
if (nd->flags & LOOKUP_RCU) {
|
|
|
|
nd->flags &= ~LOOKUP_RCU;
|
|
|
|
if (!(nd->flags & LOOKUP_ROOT))
|
|
|
|
nd->root.mnt = NULL;
|
|
|
|
spin_lock(&dentry->d_lock);
|
|
|
|
if (unlikely(!__d_rcu_to_refcount(dentry, nd->seq))) {
|
|
|
|
spin_unlock(&dentry->d_lock);
|
|
|
|
rcu_read_unlock();
|
2012-05-08 11:02:02 +07:00
|
|
|
br_read_unlock(&vfsmount_lock);
|
2011-03-25 22:00:12 +07:00
|
|
|
return -ECHILD;
|
|
|
|
}
|
|
|
|
BUG_ON(nd->inode != dentry->d_inode);
|
|
|
|
spin_unlock(&dentry->d_lock);
|
|
|
|
mntget(nd->path.mnt);
|
|
|
|
rcu_read_unlock();
|
2012-05-08 11:02:02 +07:00
|
|
|
br_read_unlock(&vfsmount_lock);
|
2011-03-25 22:00:12 +07:00
|
|
|
}
|
|
|
|
|
2011-02-23 03:50:10 +07:00
|
|
|
if (likely(!(nd->flags & LOOKUP_JUMPED)))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
if (likely(!(dentry->d_flags & DCACHE_OP_REVALIDATE)))
|
2009-12-08 00:01:50 +07:00
|
|
|
return 0;
|
|
|
|
|
2011-02-23 03:50:10 +07:00
|
|
|
if (likely(!(dentry->d_sb->s_type->fs_flags & FS_REVAL_DOT)))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/* Note: we do not d_invalidate() */
|
2011-01-07 13:49:57 +07:00
|
|
|
status = d_revalidate(dentry, nd);
|
2009-12-08 00:01:50 +07:00
|
|
|
if (status > 0)
|
|
|
|
return 0;
|
|
|
|
|
2011-02-23 03:50:10 +07:00
|
|
|
if (!status)
|
2009-12-08 00:01:50 +07:00
|
|
|
status = -ESTALE;
|
2011-02-23 03:50:10 +07:00
|
|
|
|
2011-03-25 22:00:12 +07:00
|
|
|
path_put(&nd->path);
|
2009-12-08 00:01:50 +07:00
|
|
|
return status;
|
|
|
|
}
|
|
|
|
|
2009-04-07 22:49:53 +07:00
|
|
|
static __always_inline void set_root(struct nameidata *nd)
|
|
|
|
{
|
2010-08-10 16:41:36 +07:00
|
|
|
if (!nd->root.mnt)
|
|
|
|
get_fs_root(current->fs, &nd->root);
|
2009-04-07 22:49:53 +07:00
|
|
|
}
|
|
|
|
|
2009-08-09 04:41:57 +07:00
|
|
|
static int link_path_walk(const char *, struct nameidata *);
|
|
|
|
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
static __always_inline void set_root_rcu(struct nameidata *nd)
|
|
|
|
{
|
|
|
|
if (!nd->root.mnt) {
|
|
|
|
struct fs_struct *fs = current->fs;
|
2011-01-07 13:49:53 +07:00
|
|
|
unsigned seq;
|
|
|
|
|
|
|
|
do {
|
|
|
|
seq = read_seqcount_begin(&fs->seq);
|
|
|
|
nd->root = fs->root;
|
2011-04-16 01:39:29 +07:00
|
|
|
nd->seq = __read_seqcount_begin(&nd->root.dentry->d_seq);
|
2011-01-07 13:49:53 +07:00
|
|
|
} while (read_seqcount_retry(&fs->seq, seq));
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2006-01-15 04:21:31 +07:00
|
|
|
static __always_inline int __vfs_follow_link(struct nameidata *nd, const char *link)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
int ret;
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
if (IS_ERR(link))
|
|
|
|
goto fail;
|
|
|
|
|
|
|
|
if (*link == '/') {
|
2009-04-07 22:49:53 +07:00
|
|
|
set_root(nd);
|
2008-02-15 10:34:35 +07:00
|
|
|
path_put(&nd->path);
|
2009-04-07 22:49:53 +07:00
|
|
|
nd->path = nd->root;
|
|
|
|
path_get(&nd->root);
|
2011-02-23 03:50:10 +07:00
|
|
|
nd->flags |= LOOKUP_JUMPED;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
nd->inode = nd->path.dentry->d_inode;
|
2008-11-05 21:07:21 +07:00
|
|
|
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
ret = link_path_walk(link, nd);
|
|
|
|
return ret;
|
2005-04-17 05:20:36 +07:00
|
|
|
fail:
|
2008-02-15 10:34:35 +07:00
|
|
|
path_put(&nd->path);
|
2005-04-17 05:20:36 +07:00
|
|
|
return PTR_ERR(link);
|
|
|
|
}
|
|
|
|
|
2008-02-15 10:34:35 +07:00
|
|
|
static void path_put_conditional(struct path *path, struct nameidata *nd)
|
2006-03-27 16:14:53 +07:00
|
|
|
{
|
|
|
|
dput(path->dentry);
|
2008-02-15 10:34:32 +07:00
|
|
|
if (path->mnt != nd->path.mnt)
|
2006-03-27 16:14:53 +07:00
|
|
|
mntput(path->mnt);
|
|
|
|
}
|
|
|
|
|
2011-01-14 15:42:43 +07:00
|
|
|
static inline void path_to_nameidata(const struct path *path,
|
|
|
|
struct nameidata *nd)
|
2006-03-27 16:14:53 +07:00
|
|
|
{
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
if (!(nd->flags & LOOKUP_RCU)) {
|
|
|
|
dput(nd->path.dentry);
|
|
|
|
if (nd->path.mnt != path->mnt)
|
|
|
|
mntput(nd->path.mnt);
|
2010-04-02 16:37:13 +07:00
|
|
|
}
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
nd->path.mnt = path->mnt;
|
2008-02-15 10:34:32 +07:00
|
|
|
nd->path.dentry = path->dentry;
|
2006-03-27 16:14:53 +07:00
|
|
|
}
|
|
|
|
|
2011-03-15 09:20:34 +07:00
|
|
|
static inline void put_link(struct nameidata *nd, struct path *link, void *cookie)
|
|
|
|
{
|
|
|
|
struct inode *inode = link->dentry->d_inode;
|
|
|
|
if (!IS_ERR(cookie) && inode->i_op->put_link)
|
|
|
|
inode->i_op->put_link(link->dentry, nd, cookie);
|
|
|
|
path_put(link);
|
|
|
|
}
|
|
|
|
|
2009-12-26 20:37:05 +07:00
|
|
|
static __always_inline int
|
2011-03-15 09:20:34 +07:00
|
|
|
follow_link(struct path *link, struct nameidata *nd, void **p)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
int error;
|
2011-01-14 15:42:43 +07:00
|
|
|
struct dentry *dentry = link->dentry;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2011-02-15 12:38:26 +07:00
|
|
|
BUG_ON(nd->flags & LOOKUP_RCU);
|
|
|
|
|
2011-03-16 13:45:02 +07:00
|
|
|
if (link->mnt == nd->path.mnt)
|
|
|
|
mntget(link->mnt);
|
|
|
|
|
2011-03-15 09:20:34 +07:00
|
|
|
if (unlikely(current->total_link_count >= 40)) {
|
|
|
|
*p = ERR_PTR(-ELOOP); /* no ->put_link(), please */
|
|
|
|
path_put(&nd->path);
|
|
|
|
return -ELOOP;
|
|
|
|
}
|
|
|
|
cond_resched();
|
|
|
|
current->total_link_count++;
|
|
|
|
|
2012-03-15 19:21:57 +07:00
|
|
|
touch_atime(link);
|
2005-04-17 05:20:36 +07:00
|
|
|
nd_set_link(nd, NULL);
|
2005-06-07 03:36:03 +07:00
|
|
|
|
2011-02-23 09:24:38 +07:00
|
|
|
error = security_inode_follow_link(link->dentry, nd);
|
|
|
|
if (error) {
|
|
|
|
*p = ERR_PTR(error); /* no ->put_link(), please */
|
|
|
|
path_put(&nd->path);
|
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2009-12-23 11:45:11 +07:00
|
|
|
nd->last_type = LAST_BIND;
|
2009-12-26 20:37:05 +07:00
|
|
|
*p = dentry->d_inode->i_op->follow_link(dentry, nd);
|
|
|
|
error = PTR_ERR(*p);
|
|
|
|
if (!IS_ERR(*p)) {
|
2005-04-17 05:20:36 +07:00
|
|
|
char *s = nd_get_link(nd);
|
2005-08-20 08:02:56 +07:00
|
|
|
error = 0;
|
2005-04-17 05:20:36 +07:00
|
|
|
if (s)
|
|
|
|
error = __vfs_follow_link(nd, s);
|
2011-03-14 03:42:14 +07:00
|
|
|
else if (nd->last_type == LAST_BIND) {
|
2011-02-23 03:50:10 +07:00
|
|
|
nd->flags |= LOOKUP_JUMPED;
|
2011-03-15 07:01:51 +07:00
|
|
|
nd->inode = nd->path.dentry->d_inode;
|
|
|
|
if (nd->inode->i_op->follow_link) {
|
2011-03-14 03:42:14 +07:00
|
|
|
/* stepped on a _really_ weird one */
|
|
|
|
path_put(&nd->path);
|
|
|
|
error = -ELOOP;
|
|
|
|
}
|
|
|
|
}
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
static int follow_up_rcu(struct path *path)
|
|
|
|
{
|
2011-11-25 10:19:58 +07:00
|
|
|
struct mount *mnt = real_mount(path->mnt);
|
|
|
|
struct mount *parent;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
struct dentry *mountpoint;
|
|
|
|
|
2011-11-25 10:19:58 +07:00
|
|
|
parent = mnt->mnt_parent;
|
|
|
|
if (&parent->mnt == path->mnt)
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
return 0;
|
2011-11-25 10:25:07 +07:00
|
|
|
mountpoint = mnt->mnt_mountpoint;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
path->dentry = mountpoint;
|
2011-11-25 10:19:58 +07:00
|
|
|
path->mnt = &parent->mnt;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2009-04-18 14:26:48 +07:00
|
|
|
int follow_up(struct path *path)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2011-11-25 10:19:58 +07:00
|
|
|
struct mount *mnt = real_mount(path->mnt);
|
|
|
|
struct mount *parent;
|
2005-04-17 05:20:36 +07:00
|
|
|
struct dentry *mountpoint;
|
fs: brlock vfsmount_lock
fs: brlock vfsmount_lock
Use a brlock for the vfsmount lock. It must be taken for write whenever
modifying the mount hash or associated fields, and may be taken for read when
performing mount hash lookups.
A new lock is added for the mnt-id allocator, so it doesn't need to take
the heavy vfsmount write-lock.
The number of atomics should remain the same for fastpath rlock cases, though
code would be slightly slower due to per-cpu access. Scalability is not not be
much improved in common cases yet, due to other locks (ie. dcache_lock) getting
in the way. However path lookups crossing mountpoints should be one case where
scalability is improved (currently requiring the global lock).
The slowpath is slower due to use of brlock. On a 64 core, 64 socket, 32 node
Altix system (high latency to remote nodes), a simple umount microbenchmark
(mount --bind mnt mnt2 ; umount mnt2 loop 1000 times), before this patch it
took 6.8s, afterwards took 7.1s, about 5% slower.
Cc: Al Viro <viro@ZenIV.linux.org.uk>
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2010-08-18 01:37:39 +07:00
|
|
|
|
2012-05-08 11:02:02 +07:00
|
|
|
br_read_lock(&vfsmount_lock);
|
2011-11-25 10:19:58 +07:00
|
|
|
parent = mnt->mnt_parent;
|
|
|
|
if (&parent->mnt == path->mnt) {
|
2012-05-08 11:02:02 +07:00
|
|
|
br_read_unlock(&vfsmount_lock);
|
2005-04-17 05:20:36 +07:00
|
|
|
return 0;
|
|
|
|
}
|
2011-11-25 10:19:58 +07:00
|
|
|
mntget(&parent->mnt);
|
2011-11-25 10:25:07 +07:00
|
|
|
mountpoint = dget(mnt->mnt_mountpoint);
|
2012-05-08 11:02:02 +07:00
|
|
|
br_read_unlock(&vfsmount_lock);
|
2009-04-18 14:26:48 +07:00
|
|
|
dput(path->dentry);
|
|
|
|
path->dentry = mountpoint;
|
|
|
|
mntput(path->mnt);
|
2011-11-25 10:19:58 +07:00
|
|
|
path->mnt = &parent->mnt;
|
2005-04-17 05:20:36 +07:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2011-01-07 13:49:38 +07:00
|
|
|
/*
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
* Perform an automount
|
|
|
|
* - return -EISDIR to tell follow_managed() to stop and return the path we
|
|
|
|
* were called with.
|
2005-04-17 05:20:36 +07:00
|
|
|
*/
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
static int follow_automount(struct path *path, unsigned flags,
|
|
|
|
bool *need_mntput)
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
{
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
struct vfsmount *mnt;
|
2011-01-15 02:10:03 +07:00
|
|
|
int err;
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
|
|
|
|
if (!path->dentry->d_op || !path->dentry->d_op->d_automount)
|
|
|
|
return -EREMOTE;
|
|
|
|
|
2011-09-05 23:06:26 +07:00
|
|
|
/* We don't want to mount if someone's just doing a stat -
|
|
|
|
* unless they're stat'ing a directory and appended a '/' to
|
|
|
|
* the name.
|
|
|
|
*
|
|
|
|
* We do, however, want to mount if someone wants to open or
|
|
|
|
* create a file of any type under the mountpoint, wants to
|
|
|
|
* traverse through the mountpoint or wants to open the
|
|
|
|
* mounted directory. Also, autofs may mark negative dentries
|
|
|
|
* as being automount points. These will need the attentions
|
|
|
|
* of the daemon to instantiate them before they can be used.
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
*/
|
2011-09-05 23:06:26 +07:00
|
|
|
if (!(flags & (LOOKUP_PARENT | LOOKUP_DIRECTORY |
|
vfs pathname lookup: Add LOOKUP_AUTOMOUNT flag
Since we've now turned around and made LOOKUP_FOLLOW *not* force an
automount, we want to add the ability to force an automount event on
lookup even if we don't happen to have one of the other flags that force
it implicitly (LOOKUP_OPEN, LOOKUP_DIRECTORY, LOOKUP_PARENT..)
Most cases will never want to use this, since you'd normally want to
delay automounting as long as possible, which usually implies
LOOKUP_OPEN (when we open a file or directory, we really cannot avoid
the automount any more).
But Trond argued sufficiently forcefully that at a minimum bind mounting
a file and quotactl will want to force the automount lookup. Some other
cases (like nfs_follow_remote_path()) could use it too, although
LOOKUP_DIRECTORY would work there as well.
This commit just adds the flag and logic, no users yet, though. It also
doesn't actually touch the LOOKUP_NO_AUTOMOUNT flag that is related, and
was made irrelevant by the same change that made us not follow on
LOOKUP_FOLLOW.
Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
Cc: Ian Kent <raven@themaw.net>
Cc: Jeff Layton <jlayton@redhat.com>
Cc: Miklos Szeredi <miklos@szeredi.hu>
Cc: David Howells <dhowells@redhat.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Greg KH <gregkh@suse.de>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2011-09-27 07:44:55 +07:00
|
|
|
LOOKUP_OPEN | LOOKUP_CREATE | LOOKUP_AUTOMOUNT)) &&
|
2011-09-05 23:06:26 +07:00
|
|
|
path->dentry->d_inode)
|
|
|
|
return -EISDIR;
|
|
|
|
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
current->total_link_count++;
|
|
|
|
if (current->total_link_count >= 40)
|
|
|
|
return -ELOOP;
|
|
|
|
|
|
|
|
mnt = path->dentry->d_op->d_automount(path);
|
|
|
|
if (IS_ERR(mnt)) {
|
|
|
|
/*
|
|
|
|
* The filesystem is allowed to return -EISDIR here to indicate
|
|
|
|
* it doesn't want to automount. For instance, autofs would do
|
|
|
|
* this so that its userspace daemon can mount on this dentry.
|
|
|
|
*
|
|
|
|
* However, we can only permit this if it's a terminal point in
|
|
|
|
* the path being looked up; if it wasn't then the remainder of
|
|
|
|
* the path is inaccessible and we should say so.
|
|
|
|
*/
|
2011-06-26 08:59:52 +07:00
|
|
|
if (PTR_ERR(mnt) == -EISDIR && (flags & LOOKUP_PARENT))
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
return -EREMOTE;
|
|
|
|
return PTR_ERR(mnt);
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
}
|
2011-01-15 02:10:03 +07:00
|
|
|
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
if (!mnt) /* mount collision */
|
|
|
|
return 0;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
|
VFS: Fix vfsmount overput on simultaneous automount
[Kudos to dhowells for tracking that crap down]
If two processes attempt to cause automounting on the same mountpoint at the
same time, the vfsmount holding the mountpoint will be left with one too few
references on it, causing a BUG when the kernel tries to clean up.
The problem is that lock_mount() drops the caller's reference to the
mountpoint's vfsmount in the case where it finds something already mounted on
the mountpoint as it transits to the mounted filesystem and replaces path->mnt
with the new mountpoint vfsmount.
During a pathwalk, however, we don't take a reference on the vfsmount if it is
the same as the one in the nameidata struct, but do_add_mount() doesn't know
this.
The fix is to make sure we have a ref on the vfsmount of the mountpoint before
calling do_add_mount(). However, if lock_mount() doesn't transit, we're then
left with an extra ref on the mountpoint vfsmount which needs releasing.
We can handle that in follow_managed() by not making assumptions about what
we can and what we cannot get from lookup_mnt() as the current code does.
The callers of follow_managed() expect that reference to path->mnt will be
grabbed iff path->mnt has been changed. follow_managed() and follow_automount()
keep track of whether such reference has been grabbed and assume that it'll
happen in those and only those cases that'll have us return with changed
path->mnt. That assumption is almost correct - it breaks in case of
racing automounts and in even harder to hit race between following a mountpoint
and a couple of mount --move. The thing is, we don't need to make that
assumption at all - after the end of loop in follow_manage() we can check
if path->mnt has ended up unchanged and do mntput() if needed.
The BUG can be reproduced with the following test program:
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <sys/wait.h>
int main(int argc, char **argv)
{
int pid, ws;
struct stat buf;
pid = fork();
stat(argv[1], &buf);
if (pid > 0) wait(&ws);
return 0;
}
and the following procedure:
(1) Mount an NFS volume that on the server has something else mounted on a
subdirectory. For instance, I can mount / from my server:
mount warthog:/ /mnt -t nfs4 -r
On the server /data has another filesystem mounted on it, so NFS will see
a change in FSID as it walks down the path, and will mark /mnt/data as
being a mountpoint. This will cause the automount code to be triggered.
!!! Do not look inside the mounted fs at this point !!!
(2) Run the above program on a file within the submount to generate two
simultaneous automount requests:
/tmp/forkstat /mnt/data/testfile
(3) Unmount the automounted submount:
umount /mnt/data
(4) Unmount the original mount:
umount /mnt
At this point the kernel should throw a BUG with something like the
following:
BUG: Dentry ffff880032e3c5c0{i=2,n=} still in use (1) [unmount of nfs4 0:12]
Note that the bug appears on the root dentry of the original mount, not the
mountpoint and not the submount because sys_umount() hasn't got to its final
mntput_no_expire() yet, but this isn't so obvious from the call trace:
[<ffffffff8117cd82>] shrink_dcache_for_umount+0x69/0x82
[<ffffffff8116160e>] generic_shutdown_super+0x37/0x15b
[<ffffffffa00fae56>] ? nfs_super_return_all_delegations+0x2e/0x1b1 [nfs]
[<ffffffff811617f3>] kill_anon_super+0x1d/0x7e
[<ffffffffa00d0be1>] nfs4_kill_super+0x60/0xb6 [nfs]
[<ffffffff81161c17>] deactivate_locked_super+0x34/0x83
[<ffffffff811629ff>] deactivate_super+0x6f/0x7b
[<ffffffff81186261>] mntput_no_expire+0x18d/0x199
[<ffffffff811862a8>] mntput+0x3b/0x44
[<ffffffff81186d87>] release_mounts+0xa2/0xbf
[<ffffffff811876af>] sys_umount+0x47a/0x4ba
[<ffffffff8109e1ca>] ? trace_hardirqs_on_caller+0x1fd/0x22f
[<ffffffff816ea86b>] system_call_fastpath+0x16/0x1b
as do_umount() is inlined. However, you can see release_mounts() in there.
Note also that it may be necessary to have multiple CPU cores to be able to
trigger this bug.
Tested-by: Jeff Layton <jlayton@redhat.com>
Tested-by: Ian Kent <raven@themaw.net>
Signed-off-by: David Howells <dhowells@redhat.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-06-16 21:10:06 +07:00
|
|
|
if (!*need_mntput) {
|
|
|
|
/* lock_mount() may release path->mnt on error */
|
|
|
|
mntget(path->mnt);
|
|
|
|
*need_mntput = true;
|
|
|
|
}
|
2011-01-17 13:35:23 +07:00
|
|
|
err = finish_automount(mnt, path);
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
|
2011-01-15 02:10:03 +07:00
|
|
|
switch (err) {
|
|
|
|
case -EBUSY:
|
|
|
|
/* Someone else made a mount here whilst we were busy */
|
2011-01-17 13:35:23 +07:00
|
|
|
return 0;
|
2011-01-15 02:10:03 +07:00
|
|
|
case 0:
|
VFS: Fix vfsmount overput on simultaneous automount
[Kudos to dhowells for tracking that crap down]
If two processes attempt to cause automounting on the same mountpoint at the
same time, the vfsmount holding the mountpoint will be left with one too few
references on it, causing a BUG when the kernel tries to clean up.
The problem is that lock_mount() drops the caller's reference to the
mountpoint's vfsmount in the case where it finds something already mounted on
the mountpoint as it transits to the mounted filesystem and replaces path->mnt
with the new mountpoint vfsmount.
During a pathwalk, however, we don't take a reference on the vfsmount if it is
the same as the one in the nameidata struct, but do_add_mount() doesn't know
this.
The fix is to make sure we have a ref on the vfsmount of the mountpoint before
calling do_add_mount(). However, if lock_mount() doesn't transit, we're then
left with an extra ref on the mountpoint vfsmount which needs releasing.
We can handle that in follow_managed() by not making assumptions about what
we can and what we cannot get from lookup_mnt() as the current code does.
The callers of follow_managed() expect that reference to path->mnt will be
grabbed iff path->mnt has been changed. follow_managed() and follow_automount()
keep track of whether such reference has been grabbed and assume that it'll
happen in those and only those cases that'll have us return with changed
path->mnt. That assumption is almost correct - it breaks in case of
racing automounts and in even harder to hit race between following a mountpoint
and a couple of mount --move. The thing is, we don't need to make that
assumption at all - after the end of loop in follow_manage() we can check
if path->mnt has ended up unchanged and do mntput() if needed.
The BUG can be reproduced with the following test program:
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <sys/wait.h>
int main(int argc, char **argv)
{
int pid, ws;
struct stat buf;
pid = fork();
stat(argv[1], &buf);
if (pid > 0) wait(&ws);
return 0;
}
and the following procedure:
(1) Mount an NFS volume that on the server has something else mounted on a
subdirectory. For instance, I can mount / from my server:
mount warthog:/ /mnt -t nfs4 -r
On the server /data has another filesystem mounted on it, so NFS will see
a change in FSID as it walks down the path, and will mark /mnt/data as
being a mountpoint. This will cause the automount code to be triggered.
!!! Do not look inside the mounted fs at this point !!!
(2) Run the above program on a file within the submount to generate two
simultaneous automount requests:
/tmp/forkstat /mnt/data/testfile
(3) Unmount the automounted submount:
umount /mnt/data
(4) Unmount the original mount:
umount /mnt
At this point the kernel should throw a BUG with something like the
following:
BUG: Dentry ffff880032e3c5c0{i=2,n=} still in use (1) [unmount of nfs4 0:12]
Note that the bug appears on the root dentry of the original mount, not the
mountpoint and not the submount because sys_umount() hasn't got to its final
mntput_no_expire() yet, but this isn't so obvious from the call trace:
[<ffffffff8117cd82>] shrink_dcache_for_umount+0x69/0x82
[<ffffffff8116160e>] generic_shutdown_super+0x37/0x15b
[<ffffffffa00fae56>] ? nfs_super_return_all_delegations+0x2e/0x1b1 [nfs]
[<ffffffff811617f3>] kill_anon_super+0x1d/0x7e
[<ffffffffa00d0be1>] nfs4_kill_super+0x60/0xb6 [nfs]
[<ffffffff81161c17>] deactivate_locked_super+0x34/0x83
[<ffffffff811629ff>] deactivate_super+0x6f/0x7b
[<ffffffff81186261>] mntput_no_expire+0x18d/0x199
[<ffffffff811862a8>] mntput+0x3b/0x44
[<ffffffff81186d87>] release_mounts+0xa2/0xbf
[<ffffffff811876af>] sys_umount+0x47a/0x4ba
[<ffffffff8109e1ca>] ? trace_hardirqs_on_caller+0x1fd/0x22f
[<ffffffff816ea86b>] system_call_fastpath+0x16/0x1b
as do_umount() is inlined. However, you can see release_mounts() in there.
Note also that it may be necessary to have multiple CPU cores to be able to
trigger this bug.
Tested-by: Jeff Layton <jlayton@redhat.com>
Tested-by: Ian Kent <raven@themaw.net>
Signed-off-by: David Howells <dhowells@redhat.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-06-16 21:10:06 +07:00
|
|
|
path_put(path);
|
2011-01-15 02:10:03 +07:00
|
|
|
path->mnt = mnt;
|
|
|
|
path->dentry = dget(mnt->mnt_root);
|
|
|
|
return 0;
|
2011-01-17 13:35:23 +07:00
|
|
|
default:
|
|
|
|
return err;
|
2011-01-15 02:10:03 +07:00
|
|
|
}
|
2011-01-17 13:35:23 +07:00
|
|
|
|
2005-06-07 03:36:05 +07:00
|
|
|
}
|
|
|
|
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
/*
|
|
|
|
* Handle a dentry that is managed in some way.
|
Add a dentry op to allow processes to be held during pathwalk transit
Add a dentry op (d_manage) to permit a filesystem to hold a process and make it
sleep when it tries to transit away from one of that filesystem's directories
during a pathwalk. The operation is keyed off a new dentry flag
(DCACHE_MANAGE_TRANSIT).
The filesystem is allowed to be selective about which processes it holds and
which it permits to continue on or prohibits from transiting from each flagged
directory. This will allow autofs to hold up client processes whilst letting
its userspace daemon through to maintain the directory or the stuff behind it
or mounted upon it.
The ->d_manage() dentry operation:
int (*d_manage)(struct path *path, bool mounting_here);
takes a pointer to the directory about to be transited away from and a flag
indicating whether the transit is undertaken by do_add_mount() or
do_move_mount() skipping through a pile of filesystems mounted on a mountpoint.
It should return 0 if successful and to let the process continue on its way;
-EISDIR to prohibit the caller from skipping to overmounted filesystems or
automounting, and to use this directory; or some other error code to return to
the user.
->d_manage() is called with namespace_sem writelocked if mounting_here is true
and no other locks held, so it may sleep. However, if mounting_here is true,
it may not initiate or wait for a mount or unmount upon the parameter
directory, even if the act is actually performed by userspace.
Within fs/namei.c, follow_managed() is extended to check with d_manage() first
on each managed directory, before transiting away from it or attempting to
automount upon it.
follow_down() is renamed follow_down_one() and should only be used where the
filesystem deliberately intends to avoid management steps (e.g. autofs).
A new follow_down() is added that incorporates the loop done by all other
callers of follow_down() (do_add/move_mount(), autofs and NFSD; whilst AFS, NFS
and CIFS do use it, their use is removed by converting them to use
d_automount()). The new follow_down() calls d_manage() as appropriate. It
also takes an extra parameter to indicate if it is being called from mount code
(with namespace_sem writelocked) which it passes to d_manage(). follow_down()
ignores automount points so that it can be used to mount on them.
__follow_mount_rcu() is made to abort rcu-walk mode if it hits a directory with
DCACHE_MANAGE_TRANSIT set on the basis that we're probably going to have to
sleep. It would be possible to enter d_manage() in rcu-walk mode too, and have
that determine whether to abort or not itself. That would allow the autofs
daemon to continue on in rcu-walk mode.
Note that DCACHE_MANAGE_TRANSIT on a directory should be cleared when it isn't
required as every tranist from that directory will cause d_manage() to be
invoked. It can always be set again when necessary.
==========================
WHAT THIS MEANS FOR AUTOFS
==========================
Autofs currently uses the lookup() inode op and the d_revalidate() dentry op to
trigger the automounting of indirect mounts, and both of these can be called
with i_mutex held.
autofs knows that the i_mutex will be held by the caller in lookup(), and so
can drop it before invoking the daemon - but this isn't so for d_revalidate(),
since the lock is only held on _some_ of the code paths that call it. This
means that autofs can't risk dropping i_mutex from its d_revalidate() function
before it calls the daemon.
The bug could manifest itself as, for example, a process that's trying to
validate an automount dentry that gets made to wait because that dentry is
expired and needs cleaning up:
mkdir S ffffffff8014e05a 0 32580 24956
Call Trace:
[<ffffffff885371fd>] :autofs4:autofs4_wait+0x674/0x897
[<ffffffff80127f7d>] avc_has_perm+0x46/0x58
[<ffffffff8009fdcf>] autoremove_wake_function+0x0/0x2e
[<ffffffff88537be6>] :autofs4:autofs4_expire_wait+0x41/0x6b
[<ffffffff88535cfc>] :autofs4:autofs4_revalidate+0x91/0x149
[<ffffffff80036d96>] __lookup_hash+0xa0/0x12f
[<ffffffff80057a2f>] lookup_create+0x46/0x80
[<ffffffff800e6e31>] sys_mkdirat+0x56/0xe4
versus the automount daemon which wants to remove that dentry, but can't
because the normal process is holding the i_mutex lock:
automount D ffffffff8014e05a 0 32581 1 32561
Call Trace:
[<ffffffff80063c3f>] __mutex_lock_slowpath+0x60/0x9b
[<ffffffff8000ccf1>] do_path_lookup+0x2ca/0x2f1
[<ffffffff80063c89>] .text.lock.mutex+0xf/0x14
[<ffffffff800e6d55>] do_rmdir+0x77/0xde
[<ffffffff8005d229>] tracesys+0x71/0xe0
[<ffffffff8005d28d>] tracesys+0xd5/0xe0
which means that the system is deadlocked.
This patch allows autofs to hold up normal processes whilst the daemon goes
ahead and does things to the dentry tree behind the automouter point without
risking a deadlock as almost no locks are held in d_manage() and none in
d_automount().
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:26 +07:00
|
|
|
* - Flagged for transit management (autofs)
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
* - Flagged as mountpoint
|
|
|
|
* - Flagged as automount point
|
|
|
|
*
|
|
|
|
* This may only be called in refwalk mode.
|
|
|
|
*
|
|
|
|
* Serialization is taken care of in namespace.c
|
|
|
|
*/
|
|
|
|
static int follow_managed(struct path *path, unsigned flags)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
VFS: Fix vfsmount overput on simultaneous automount
[Kudos to dhowells for tracking that crap down]
If two processes attempt to cause automounting on the same mountpoint at the
same time, the vfsmount holding the mountpoint will be left with one too few
references on it, causing a BUG when the kernel tries to clean up.
The problem is that lock_mount() drops the caller's reference to the
mountpoint's vfsmount in the case where it finds something already mounted on
the mountpoint as it transits to the mounted filesystem and replaces path->mnt
with the new mountpoint vfsmount.
During a pathwalk, however, we don't take a reference on the vfsmount if it is
the same as the one in the nameidata struct, but do_add_mount() doesn't know
this.
The fix is to make sure we have a ref on the vfsmount of the mountpoint before
calling do_add_mount(). However, if lock_mount() doesn't transit, we're then
left with an extra ref on the mountpoint vfsmount which needs releasing.
We can handle that in follow_managed() by not making assumptions about what
we can and what we cannot get from lookup_mnt() as the current code does.
The callers of follow_managed() expect that reference to path->mnt will be
grabbed iff path->mnt has been changed. follow_managed() and follow_automount()
keep track of whether such reference has been grabbed and assume that it'll
happen in those and only those cases that'll have us return with changed
path->mnt. That assumption is almost correct - it breaks in case of
racing automounts and in even harder to hit race between following a mountpoint
and a couple of mount --move. The thing is, we don't need to make that
assumption at all - after the end of loop in follow_manage() we can check
if path->mnt has ended up unchanged and do mntput() if needed.
The BUG can be reproduced with the following test program:
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <sys/wait.h>
int main(int argc, char **argv)
{
int pid, ws;
struct stat buf;
pid = fork();
stat(argv[1], &buf);
if (pid > 0) wait(&ws);
return 0;
}
and the following procedure:
(1) Mount an NFS volume that on the server has something else mounted on a
subdirectory. For instance, I can mount / from my server:
mount warthog:/ /mnt -t nfs4 -r
On the server /data has another filesystem mounted on it, so NFS will see
a change in FSID as it walks down the path, and will mark /mnt/data as
being a mountpoint. This will cause the automount code to be triggered.
!!! Do not look inside the mounted fs at this point !!!
(2) Run the above program on a file within the submount to generate two
simultaneous automount requests:
/tmp/forkstat /mnt/data/testfile
(3) Unmount the automounted submount:
umount /mnt/data
(4) Unmount the original mount:
umount /mnt
At this point the kernel should throw a BUG with something like the
following:
BUG: Dentry ffff880032e3c5c0{i=2,n=} still in use (1) [unmount of nfs4 0:12]
Note that the bug appears on the root dentry of the original mount, not the
mountpoint and not the submount because sys_umount() hasn't got to its final
mntput_no_expire() yet, but this isn't so obvious from the call trace:
[<ffffffff8117cd82>] shrink_dcache_for_umount+0x69/0x82
[<ffffffff8116160e>] generic_shutdown_super+0x37/0x15b
[<ffffffffa00fae56>] ? nfs_super_return_all_delegations+0x2e/0x1b1 [nfs]
[<ffffffff811617f3>] kill_anon_super+0x1d/0x7e
[<ffffffffa00d0be1>] nfs4_kill_super+0x60/0xb6 [nfs]
[<ffffffff81161c17>] deactivate_locked_super+0x34/0x83
[<ffffffff811629ff>] deactivate_super+0x6f/0x7b
[<ffffffff81186261>] mntput_no_expire+0x18d/0x199
[<ffffffff811862a8>] mntput+0x3b/0x44
[<ffffffff81186d87>] release_mounts+0xa2/0xbf
[<ffffffff811876af>] sys_umount+0x47a/0x4ba
[<ffffffff8109e1ca>] ? trace_hardirqs_on_caller+0x1fd/0x22f
[<ffffffff816ea86b>] system_call_fastpath+0x16/0x1b
as do_umount() is inlined. However, you can see release_mounts() in there.
Note also that it may be necessary to have multiple CPU cores to be able to
trigger this bug.
Tested-by: Jeff Layton <jlayton@redhat.com>
Tested-by: Ian Kent <raven@themaw.net>
Signed-off-by: David Howells <dhowells@redhat.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-06-16 21:10:06 +07:00
|
|
|
struct vfsmount *mnt = path->mnt; /* held by caller, must be left alone */
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
unsigned managed;
|
|
|
|
bool need_mntput = false;
|
VFS: Fix vfsmount overput on simultaneous automount
[Kudos to dhowells for tracking that crap down]
If two processes attempt to cause automounting on the same mountpoint at the
same time, the vfsmount holding the mountpoint will be left with one too few
references on it, causing a BUG when the kernel tries to clean up.
The problem is that lock_mount() drops the caller's reference to the
mountpoint's vfsmount in the case where it finds something already mounted on
the mountpoint as it transits to the mounted filesystem and replaces path->mnt
with the new mountpoint vfsmount.
During a pathwalk, however, we don't take a reference on the vfsmount if it is
the same as the one in the nameidata struct, but do_add_mount() doesn't know
this.
The fix is to make sure we have a ref on the vfsmount of the mountpoint before
calling do_add_mount(). However, if lock_mount() doesn't transit, we're then
left with an extra ref on the mountpoint vfsmount which needs releasing.
We can handle that in follow_managed() by not making assumptions about what
we can and what we cannot get from lookup_mnt() as the current code does.
The callers of follow_managed() expect that reference to path->mnt will be
grabbed iff path->mnt has been changed. follow_managed() and follow_automount()
keep track of whether such reference has been grabbed and assume that it'll
happen in those and only those cases that'll have us return with changed
path->mnt. That assumption is almost correct - it breaks in case of
racing automounts and in even harder to hit race between following a mountpoint
and a couple of mount --move. The thing is, we don't need to make that
assumption at all - after the end of loop in follow_manage() we can check
if path->mnt has ended up unchanged and do mntput() if needed.
The BUG can be reproduced with the following test program:
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <sys/wait.h>
int main(int argc, char **argv)
{
int pid, ws;
struct stat buf;
pid = fork();
stat(argv[1], &buf);
if (pid > 0) wait(&ws);
return 0;
}
and the following procedure:
(1) Mount an NFS volume that on the server has something else mounted on a
subdirectory. For instance, I can mount / from my server:
mount warthog:/ /mnt -t nfs4 -r
On the server /data has another filesystem mounted on it, so NFS will see
a change in FSID as it walks down the path, and will mark /mnt/data as
being a mountpoint. This will cause the automount code to be triggered.
!!! Do not look inside the mounted fs at this point !!!
(2) Run the above program on a file within the submount to generate two
simultaneous automount requests:
/tmp/forkstat /mnt/data/testfile
(3) Unmount the automounted submount:
umount /mnt/data
(4) Unmount the original mount:
umount /mnt
At this point the kernel should throw a BUG with something like the
following:
BUG: Dentry ffff880032e3c5c0{i=2,n=} still in use (1) [unmount of nfs4 0:12]
Note that the bug appears on the root dentry of the original mount, not the
mountpoint and not the submount because sys_umount() hasn't got to its final
mntput_no_expire() yet, but this isn't so obvious from the call trace:
[<ffffffff8117cd82>] shrink_dcache_for_umount+0x69/0x82
[<ffffffff8116160e>] generic_shutdown_super+0x37/0x15b
[<ffffffffa00fae56>] ? nfs_super_return_all_delegations+0x2e/0x1b1 [nfs]
[<ffffffff811617f3>] kill_anon_super+0x1d/0x7e
[<ffffffffa00d0be1>] nfs4_kill_super+0x60/0xb6 [nfs]
[<ffffffff81161c17>] deactivate_locked_super+0x34/0x83
[<ffffffff811629ff>] deactivate_super+0x6f/0x7b
[<ffffffff81186261>] mntput_no_expire+0x18d/0x199
[<ffffffff811862a8>] mntput+0x3b/0x44
[<ffffffff81186d87>] release_mounts+0xa2/0xbf
[<ffffffff811876af>] sys_umount+0x47a/0x4ba
[<ffffffff8109e1ca>] ? trace_hardirqs_on_caller+0x1fd/0x22f
[<ffffffff816ea86b>] system_call_fastpath+0x16/0x1b
as do_umount() is inlined. However, you can see release_mounts() in there.
Note also that it may be necessary to have multiple CPU cores to be able to
trigger this bug.
Tested-by: Jeff Layton <jlayton@redhat.com>
Tested-by: Ian Kent <raven@themaw.net>
Signed-off-by: David Howells <dhowells@redhat.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-06-16 21:10:06 +07:00
|
|
|
int ret = 0;
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
|
|
|
|
/* Given that we're not holding a lock here, we retain the value in a
|
|
|
|
* local variable for each dentry as we look at it so that we don't see
|
|
|
|
* the components of that value change under us */
|
|
|
|
while (managed = ACCESS_ONCE(path->dentry->d_flags),
|
|
|
|
managed &= DCACHE_MANAGED_DENTRY,
|
|
|
|
unlikely(managed != 0)) {
|
Add a dentry op to allow processes to be held during pathwalk transit
Add a dentry op (d_manage) to permit a filesystem to hold a process and make it
sleep when it tries to transit away from one of that filesystem's directories
during a pathwalk. The operation is keyed off a new dentry flag
(DCACHE_MANAGE_TRANSIT).
The filesystem is allowed to be selective about which processes it holds and
which it permits to continue on or prohibits from transiting from each flagged
directory. This will allow autofs to hold up client processes whilst letting
its userspace daemon through to maintain the directory or the stuff behind it
or mounted upon it.
The ->d_manage() dentry operation:
int (*d_manage)(struct path *path, bool mounting_here);
takes a pointer to the directory about to be transited away from and a flag
indicating whether the transit is undertaken by do_add_mount() or
do_move_mount() skipping through a pile of filesystems mounted on a mountpoint.
It should return 0 if successful and to let the process continue on its way;
-EISDIR to prohibit the caller from skipping to overmounted filesystems or
automounting, and to use this directory; or some other error code to return to
the user.
->d_manage() is called with namespace_sem writelocked if mounting_here is true
and no other locks held, so it may sleep. However, if mounting_here is true,
it may not initiate or wait for a mount or unmount upon the parameter
directory, even if the act is actually performed by userspace.
Within fs/namei.c, follow_managed() is extended to check with d_manage() first
on each managed directory, before transiting away from it or attempting to
automount upon it.
follow_down() is renamed follow_down_one() and should only be used where the
filesystem deliberately intends to avoid management steps (e.g. autofs).
A new follow_down() is added that incorporates the loop done by all other
callers of follow_down() (do_add/move_mount(), autofs and NFSD; whilst AFS, NFS
and CIFS do use it, their use is removed by converting them to use
d_automount()). The new follow_down() calls d_manage() as appropriate. It
also takes an extra parameter to indicate if it is being called from mount code
(with namespace_sem writelocked) which it passes to d_manage(). follow_down()
ignores automount points so that it can be used to mount on them.
__follow_mount_rcu() is made to abort rcu-walk mode if it hits a directory with
DCACHE_MANAGE_TRANSIT set on the basis that we're probably going to have to
sleep. It would be possible to enter d_manage() in rcu-walk mode too, and have
that determine whether to abort or not itself. That would allow the autofs
daemon to continue on in rcu-walk mode.
Note that DCACHE_MANAGE_TRANSIT on a directory should be cleared when it isn't
required as every tranist from that directory will cause d_manage() to be
invoked. It can always be set again when necessary.
==========================
WHAT THIS MEANS FOR AUTOFS
==========================
Autofs currently uses the lookup() inode op and the d_revalidate() dentry op to
trigger the automounting of indirect mounts, and both of these can be called
with i_mutex held.
autofs knows that the i_mutex will be held by the caller in lookup(), and so
can drop it before invoking the daemon - but this isn't so for d_revalidate(),
since the lock is only held on _some_ of the code paths that call it. This
means that autofs can't risk dropping i_mutex from its d_revalidate() function
before it calls the daemon.
The bug could manifest itself as, for example, a process that's trying to
validate an automount dentry that gets made to wait because that dentry is
expired and needs cleaning up:
mkdir S ffffffff8014e05a 0 32580 24956
Call Trace:
[<ffffffff885371fd>] :autofs4:autofs4_wait+0x674/0x897
[<ffffffff80127f7d>] avc_has_perm+0x46/0x58
[<ffffffff8009fdcf>] autoremove_wake_function+0x0/0x2e
[<ffffffff88537be6>] :autofs4:autofs4_expire_wait+0x41/0x6b
[<ffffffff88535cfc>] :autofs4:autofs4_revalidate+0x91/0x149
[<ffffffff80036d96>] __lookup_hash+0xa0/0x12f
[<ffffffff80057a2f>] lookup_create+0x46/0x80
[<ffffffff800e6e31>] sys_mkdirat+0x56/0xe4
versus the automount daemon which wants to remove that dentry, but can't
because the normal process is holding the i_mutex lock:
automount D ffffffff8014e05a 0 32581 1 32561
Call Trace:
[<ffffffff80063c3f>] __mutex_lock_slowpath+0x60/0x9b
[<ffffffff8000ccf1>] do_path_lookup+0x2ca/0x2f1
[<ffffffff80063c89>] .text.lock.mutex+0xf/0x14
[<ffffffff800e6d55>] do_rmdir+0x77/0xde
[<ffffffff8005d229>] tracesys+0x71/0xe0
[<ffffffff8005d28d>] tracesys+0xd5/0xe0
which means that the system is deadlocked.
This patch allows autofs to hold up normal processes whilst the daemon goes
ahead and does things to the dentry tree behind the automouter point without
risking a deadlock as almost no locks are held in d_manage() and none in
d_automount().
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:26 +07:00
|
|
|
/* Allow the filesystem to manage the transit without i_mutex
|
|
|
|
* being held. */
|
|
|
|
if (managed & DCACHE_MANAGE_TRANSIT) {
|
|
|
|
BUG_ON(!path->dentry->d_op);
|
|
|
|
BUG_ON(!path->dentry->d_op->d_manage);
|
2011-03-18 20:09:02 +07:00
|
|
|
ret = path->dentry->d_op->d_manage(path->dentry, false);
|
Add a dentry op to allow processes to be held during pathwalk transit
Add a dentry op (d_manage) to permit a filesystem to hold a process and make it
sleep when it tries to transit away from one of that filesystem's directories
during a pathwalk. The operation is keyed off a new dentry flag
(DCACHE_MANAGE_TRANSIT).
The filesystem is allowed to be selective about which processes it holds and
which it permits to continue on or prohibits from transiting from each flagged
directory. This will allow autofs to hold up client processes whilst letting
its userspace daemon through to maintain the directory or the stuff behind it
or mounted upon it.
The ->d_manage() dentry operation:
int (*d_manage)(struct path *path, bool mounting_here);
takes a pointer to the directory about to be transited away from and a flag
indicating whether the transit is undertaken by do_add_mount() or
do_move_mount() skipping through a pile of filesystems mounted on a mountpoint.
It should return 0 if successful and to let the process continue on its way;
-EISDIR to prohibit the caller from skipping to overmounted filesystems or
automounting, and to use this directory; or some other error code to return to
the user.
->d_manage() is called with namespace_sem writelocked if mounting_here is true
and no other locks held, so it may sleep. However, if mounting_here is true,
it may not initiate or wait for a mount or unmount upon the parameter
directory, even if the act is actually performed by userspace.
Within fs/namei.c, follow_managed() is extended to check with d_manage() first
on each managed directory, before transiting away from it or attempting to
automount upon it.
follow_down() is renamed follow_down_one() and should only be used where the
filesystem deliberately intends to avoid management steps (e.g. autofs).
A new follow_down() is added that incorporates the loop done by all other
callers of follow_down() (do_add/move_mount(), autofs and NFSD; whilst AFS, NFS
and CIFS do use it, their use is removed by converting them to use
d_automount()). The new follow_down() calls d_manage() as appropriate. It
also takes an extra parameter to indicate if it is being called from mount code
(with namespace_sem writelocked) which it passes to d_manage(). follow_down()
ignores automount points so that it can be used to mount on them.
__follow_mount_rcu() is made to abort rcu-walk mode if it hits a directory with
DCACHE_MANAGE_TRANSIT set on the basis that we're probably going to have to
sleep. It would be possible to enter d_manage() in rcu-walk mode too, and have
that determine whether to abort or not itself. That would allow the autofs
daemon to continue on in rcu-walk mode.
Note that DCACHE_MANAGE_TRANSIT on a directory should be cleared when it isn't
required as every tranist from that directory will cause d_manage() to be
invoked. It can always be set again when necessary.
==========================
WHAT THIS MEANS FOR AUTOFS
==========================
Autofs currently uses the lookup() inode op and the d_revalidate() dentry op to
trigger the automounting of indirect mounts, and both of these can be called
with i_mutex held.
autofs knows that the i_mutex will be held by the caller in lookup(), and so
can drop it before invoking the daemon - but this isn't so for d_revalidate(),
since the lock is only held on _some_ of the code paths that call it. This
means that autofs can't risk dropping i_mutex from its d_revalidate() function
before it calls the daemon.
The bug could manifest itself as, for example, a process that's trying to
validate an automount dentry that gets made to wait because that dentry is
expired and needs cleaning up:
mkdir S ffffffff8014e05a 0 32580 24956
Call Trace:
[<ffffffff885371fd>] :autofs4:autofs4_wait+0x674/0x897
[<ffffffff80127f7d>] avc_has_perm+0x46/0x58
[<ffffffff8009fdcf>] autoremove_wake_function+0x0/0x2e
[<ffffffff88537be6>] :autofs4:autofs4_expire_wait+0x41/0x6b
[<ffffffff88535cfc>] :autofs4:autofs4_revalidate+0x91/0x149
[<ffffffff80036d96>] __lookup_hash+0xa0/0x12f
[<ffffffff80057a2f>] lookup_create+0x46/0x80
[<ffffffff800e6e31>] sys_mkdirat+0x56/0xe4
versus the automount daemon which wants to remove that dentry, but can't
because the normal process is holding the i_mutex lock:
automount D ffffffff8014e05a 0 32581 1 32561
Call Trace:
[<ffffffff80063c3f>] __mutex_lock_slowpath+0x60/0x9b
[<ffffffff8000ccf1>] do_path_lookup+0x2ca/0x2f1
[<ffffffff80063c89>] .text.lock.mutex+0xf/0x14
[<ffffffff800e6d55>] do_rmdir+0x77/0xde
[<ffffffff8005d229>] tracesys+0x71/0xe0
[<ffffffff8005d28d>] tracesys+0xd5/0xe0
which means that the system is deadlocked.
This patch allows autofs to hold up normal processes whilst the daemon goes
ahead and does things to the dentry tree behind the automouter point without
risking a deadlock as almost no locks are held in d_manage() and none in
d_automount().
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:26 +07:00
|
|
|
if (ret < 0)
|
VFS: Fix vfsmount overput on simultaneous automount
[Kudos to dhowells for tracking that crap down]
If two processes attempt to cause automounting on the same mountpoint at the
same time, the vfsmount holding the mountpoint will be left with one too few
references on it, causing a BUG when the kernel tries to clean up.
The problem is that lock_mount() drops the caller's reference to the
mountpoint's vfsmount in the case where it finds something already mounted on
the mountpoint as it transits to the mounted filesystem and replaces path->mnt
with the new mountpoint vfsmount.
During a pathwalk, however, we don't take a reference on the vfsmount if it is
the same as the one in the nameidata struct, but do_add_mount() doesn't know
this.
The fix is to make sure we have a ref on the vfsmount of the mountpoint before
calling do_add_mount(). However, if lock_mount() doesn't transit, we're then
left with an extra ref on the mountpoint vfsmount which needs releasing.
We can handle that in follow_managed() by not making assumptions about what
we can and what we cannot get from lookup_mnt() as the current code does.
The callers of follow_managed() expect that reference to path->mnt will be
grabbed iff path->mnt has been changed. follow_managed() and follow_automount()
keep track of whether such reference has been grabbed and assume that it'll
happen in those and only those cases that'll have us return with changed
path->mnt. That assumption is almost correct - it breaks in case of
racing automounts and in even harder to hit race between following a mountpoint
and a couple of mount --move. The thing is, we don't need to make that
assumption at all - after the end of loop in follow_manage() we can check
if path->mnt has ended up unchanged and do mntput() if needed.
The BUG can be reproduced with the following test program:
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <sys/wait.h>
int main(int argc, char **argv)
{
int pid, ws;
struct stat buf;
pid = fork();
stat(argv[1], &buf);
if (pid > 0) wait(&ws);
return 0;
}
and the following procedure:
(1) Mount an NFS volume that on the server has something else mounted on a
subdirectory. For instance, I can mount / from my server:
mount warthog:/ /mnt -t nfs4 -r
On the server /data has another filesystem mounted on it, so NFS will see
a change in FSID as it walks down the path, and will mark /mnt/data as
being a mountpoint. This will cause the automount code to be triggered.
!!! Do not look inside the mounted fs at this point !!!
(2) Run the above program on a file within the submount to generate two
simultaneous automount requests:
/tmp/forkstat /mnt/data/testfile
(3) Unmount the automounted submount:
umount /mnt/data
(4) Unmount the original mount:
umount /mnt
At this point the kernel should throw a BUG with something like the
following:
BUG: Dentry ffff880032e3c5c0{i=2,n=} still in use (1) [unmount of nfs4 0:12]
Note that the bug appears on the root dentry of the original mount, not the
mountpoint and not the submount because sys_umount() hasn't got to its final
mntput_no_expire() yet, but this isn't so obvious from the call trace:
[<ffffffff8117cd82>] shrink_dcache_for_umount+0x69/0x82
[<ffffffff8116160e>] generic_shutdown_super+0x37/0x15b
[<ffffffffa00fae56>] ? nfs_super_return_all_delegations+0x2e/0x1b1 [nfs]
[<ffffffff811617f3>] kill_anon_super+0x1d/0x7e
[<ffffffffa00d0be1>] nfs4_kill_super+0x60/0xb6 [nfs]
[<ffffffff81161c17>] deactivate_locked_super+0x34/0x83
[<ffffffff811629ff>] deactivate_super+0x6f/0x7b
[<ffffffff81186261>] mntput_no_expire+0x18d/0x199
[<ffffffff811862a8>] mntput+0x3b/0x44
[<ffffffff81186d87>] release_mounts+0xa2/0xbf
[<ffffffff811876af>] sys_umount+0x47a/0x4ba
[<ffffffff8109e1ca>] ? trace_hardirqs_on_caller+0x1fd/0x22f
[<ffffffff816ea86b>] system_call_fastpath+0x16/0x1b
as do_umount() is inlined. However, you can see release_mounts() in there.
Note also that it may be necessary to have multiple CPU cores to be able to
trigger this bug.
Tested-by: Jeff Layton <jlayton@redhat.com>
Tested-by: Ian Kent <raven@themaw.net>
Signed-off-by: David Howells <dhowells@redhat.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-06-16 21:10:06 +07:00
|
|
|
break;
|
Add a dentry op to allow processes to be held during pathwalk transit
Add a dentry op (d_manage) to permit a filesystem to hold a process and make it
sleep when it tries to transit away from one of that filesystem's directories
during a pathwalk. The operation is keyed off a new dentry flag
(DCACHE_MANAGE_TRANSIT).
The filesystem is allowed to be selective about which processes it holds and
which it permits to continue on or prohibits from transiting from each flagged
directory. This will allow autofs to hold up client processes whilst letting
its userspace daemon through to maintain the directory or the stuff behind it
or mounted upon it.
The ->d_manage() dentry operation:
int (*d_manage)(struct path *path, bool mounting_here);
takes a pointer to the directory about to be transited away from and a flag
indicating whether the transit is undertaken by do_add_mount() or
do_move_mount() skipping through a pile of filesystems mounted on a mountpoint.
It should return 0 if successful and to let the process continue on its way;
-EISDIR to prohibit the caller from skipping to overmounted filesystems or
automounting, and to use this directory; or some other error code to return to
the user.
->d_manage() is called with namespace_sem writelocked if mounting_here is true
and no other locks held, so it may sleep. However, if mounting_here is true,
it may not initiate or wait for a mount or unmount upon the parameter
directory, even if the act is actually performed by userspace.
Within fs/namei.c, follow_managed() is extended to check with d_manage() first
on each managed directory, before transiting away from it or attempting to
automount upon it.
follow_down() is renamed follow_down_one() and should only be used where the
filesystem deliberately intends to avoid management steps (e.g. autofs).
A new follow_down() is added that incorporates the loop done by all other
callers of follow_down() (do_add/move_mount(), autofs and NFSD; whilst AFS, NFS
and CIFS do use it, their use is removed by converting them to use
d_automount()). The new follow_down() calls d_manage() as appropriate. It
also takes an extra parameter to indicate if it is being called from mount code
(with namespace_sem writelocked) which it passes to d_manage(). follow_down()
ignores automount points so that it can be used to mount on them.
__follow_mount_rcu() is made to abort rcu-walk mode if it hits a directory with
DCACHE_MANAGE_TRANSIT set on the basis that we're probably going to have to
sleep. It would be possible to enter d_manage() in rcu-walk mode too, and have
that determine whether to abort or not itself. That would allow the autofs
daemon to continue on in rcu-walk mode.
Note that DCACHE_MANAGE_TRANSIT on a directory should be cleared when it isn't
required as every tranist from that directory will cause d_manage() to be
invoked. It can always be set again when necessary.
==========================
WHAT THIS MEANS FOR AUTOFS
==========================
Autofs currently uses the lookup() inode op and the d_revalidate() dentry op to
trigger the automounting of indirect mounts, and both of these can be called
with i_mutex held.
autofs knows that the i_mutex will be held by the caller in lookup(), and so
can drop it before invoking the daemon - but this isn't so for d_revalidate(),
since the lock is only held on _some_ of the code paths that call it. This
means that autofs can't risk dropping i_mutex from its d_revalidate() function
before it calls the daemon.
The bug could manifest itself as, for example, a process that's trying to
validate an automount dentry that gets made to wait because that dentry is
expired and needs cleaning up:
mkdir S ffffffff8014e05a 0 32580 24956
Call Trace:
[<ffffffff885371fd>] :autofs4:autofs4_wait+0x674/0x897
[<ffffffff80127f7d>] avc_has_perm+0x46/0x58
[<ffffffff8009fdcf>] autoremove_wake_function+0x0/0x2e
[<ffffffff88537be6>] :autofs4:autofs4_expire_wait+0x41/0x6b
[<ffffffff88535cfc>] :autofs4:autofs4_revalidate+0x91/0x149
[<ffffffff80036d96>] __lookup_hash+0xa0/0x12f
[<ffffffff80057a2f>] lookup_create+0x46/0x80
[<ffffffff800e6e31>] sys_mkdirat+0x56/0xe4
versus the automount daemon which wants to remove that dentry, but can't
because the normal process is holding the i_mutex lock:
automount D ffffffff8014e05a 0 32581 1 32561
Call Trace:
[<ffffffff80063c3f>] __mutex_lock_slowpath+0x60/0x9b
[<ffffffff8000ccf1>] do_path_lookup+0x2ca/0x2f1
[<ffffffff80063c89>] .text.lock.mutex+0xf/0x14
[<ffffffff800e6d55>] do_rmdir+0x77/0xde
[<ffffffff8005d229>] tracesys+0x71/0xe0
[<ffffffff8005d28d>] tracesys+0xd5/0xe0
which means that the system is deadlocked.
This patch allows autofs to hold up normal processes whilst the daemon goes
ahead and does things to the dentry tree behind the automouter point without
risking a deadlock as almost no locks are held in d_manage() and none in
d_automount().
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:26 +07:00
|
|
|
}
|
|
|
|
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
/* Transit to a mounted filesystem. */
|
|
|
|
if (managed & DCACHE_MOUNTED) {
|
|
|
|
struct vfsmount *mounted = lookup_mnt(path);
|
|
|
|
if (mounted) {
|
|
|
|
dput(path->dentry);
|
|
|
|
if (need_mntput)
|
|
|
|
mntput(path->mnt);
|
|
|
|
path->mnt = mounted;
|
|
|
|
path->dentry = dget(mounted->mnt_root);
|
|
|
|
need_mntput = true;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Something is mounted on this dentry in another
|
|
|
|
* namespace and/or whatever was mounted there in this
|
|
|
|
* namespace got unmounted before we managed to get the
|
|
|
|
* vfsmount_lock */
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Handle an automount point */
|
|
|
|
if (managed & DCACHE_NEED_AUTOMOUNT) {
|
|
|
|
ret = follow_automount(path, flags, &need_mntput);
|
|
|
|
if (ret < 0)
|
VFS: Fix vfsmount overput on simultaneous automount
[Kudos to dhowells for tracking that crap down]
If two processes attempt to cause automounting on the same mountpoint at the
same time, the vfsmount holding the mountpoint will be left with one too few
references on it, causing a BUG when the kernel tries to clean up.
The problem is that lock_mount() drops the caller's reference to the
mountpoint's vfsmount in the case where it finds something already mounted on
the mountpoint as it transits to the mounted filesystem and replaces path->mnt
with the new mountpoint vfsmount.
During a pathwalk, however, we don't take a reference on the vfsmount if it is
the same as the one in the nameidata struct, but do_add_mount() doesn't know
this.
The fix is to make sure we have a ref on the vfsmount of the mountpoint before
calling do_add_mount(). However, if lock_mount() doesn't transit, we're then
left with an extra ref on the mountpoint vfsmount which needs releasing.
We can handle that in follow_managed() by not making assumptions about what
we can and what we cannot get from lookup_mnt() as the current code does.
The callers of follow_managed() expect that reference to path->mnt will be
grabbed iff path->mnt has been changed. follow_managed() and follow_automount()
keep track of whether such reference has been grabbed and assume that it'll
happen in those and only those cases that'll have us return with changed
path->mnt. That assumption is almost correct - it breaks in case of
racing automounts and in even harder to hit race between following a mountpoint
and a couple of mount --move. The thing is, we don't need to make that
assumption at all - after the end of loop in follow_manage() we can check
if path->mnt has ended up unchanged and do mntput() if needed.
The BUG can be reproduced with the following test program:
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <sys/wait.h>
int main(int argc, char **argv)
{
int pid, ws;
struct stat buf;
pid = fork();
stat(argv[1], &buf);
if (pid > 0) wait(&ws);
return 0;
}
and the following procedure:
(1) Mount an NFS volume that on the server has something else mounted on a
subdirectory. For instance, I can mount / from my server:
mount warthog:/ /mnt -t nfs4 -r
On the server /data has another filesystem mounted on it, so NFS will see
a change in FSID as it walks down the path, and will mark /mnt/data as
being a mountpoint. This will cause the automount code to be triggered.
!!! Do not look inside the mounted fs at this point !!!
(2) Run the above program on a file within the submount to generate two
simultaneous automount requests:
/tmp/forkstat /mnt/data/testfile
(3) Unmount the automounted submount:
umount /mnt/data
(4) Unmount the original mount:
umount /mnt
At this point the kernel should throw a BUG with something like the
following:
BUG: Dentry ffff880032e3c5c0{i=2,n=} still in use (1) [unmount of nfs4 0:12]
Note that the bug appears on the root dentry of the original mount, not the
mountpoint and not the submount because sys_umount() hasn't got to its final
mntput_no_expire() yet, but this isn't so obvious from the call trace:
[<ffffffff8117cd82>] shrink_dcache_for_umount+0x69/0x82
[<ffffffff8116160e>] generic_shutdown_super+0x37/0x15b
[<ffffffffa00fae56>] ? nfs_super_return_all_delegations+0x2e/0x1b1 [nfs]
[<ffffffff811617f3>] kill_anon_super+0x1d/0x7e
[<ffffffffa00d0be1>] nfs4_kill_super+0x60/0xb6 [nfs]
[<ffffffff81161c17>] deactivate_locked_super+0x34/0x83
[<ffffffff811629ff>] deactivate_super+0x6f/0x7b
[<ffffffff81186261>] mntput_no_expire+0x18d/0x199
[<ffffffff811862a8>] mntput+0x3b/0x44
[<ffffffff81186d87>] release_mounts+0xa2/0xbf
[<ffffffff811876af>] sys_umount+0x47a/0x4ba
[<ffffffff8109e1ca>] ? trace_hardirqs_on_caller+0x1fd/0x22f
[<ffffffff816ea86b>] system_call_fastpath+0x16/0x1b
as do_umount() is inlined. However, you can see release_mounts() in there.
Note also that it may be necessary to have multiple CPU cores to be able to
trigger this bug.
Tested-by: Jeff Layton <jlayton@redhat.com>
Tested-by: Ian Kent <raven@themaw.net>
Signed-off-by: David Howells <dhowells@redhat.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-06-16 21:10:06 +07:00
|
|
|
break;
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* We didn't change the current path point */
|
|
|
|
break;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
VFS: Fix vfsmount overput on simultaneous automount
[Kudos to dhowells for tracking that crap down]
If two processes attempt to cause automounting on the same mountpoint at the
same time, the vfsmount holding the mountpoint will be left with one too few
references on it, causing a BUG when the kernel tries to clean up.
The problem is that lock_mount() drops the caller's reference to the
mountpoint's vfsmount in the case where it finds something already mounted on
the mountpoint as it transits to the mounted filesystem and replaces path->mnt
with the new mountpoint vfsmount.
During a pathwalk, however, we don't take a reference on the vfsmount if it is
the same as the one in the nameidata struct, but do_add_mount() doesn't know
this.
The fix is to make sure we have a ref on the vfsmount of the mountpoint before
calling do_add_mount(). However, if lock_mount() doesn't transit, we're then
left with an extra ref on the mountpoint vfsmount which needs releasing.
We can handle that in follow_managed() by not making assumptions about what
we can and what we cannot get from lookup_mnt() as the current code does.
The callers of follow_managed() expect that reference to path->mnt will be
grabbed iff path->mnt has been changed. follow_managed() and follow_automount()
keep track of whether such reference has been grabbed and assume that it'll
happen in those and only those cases that'll have us return with changed
path->mnt. That assumption is almost correct - it breaks in case of
racing automounts and in even harder to hit race between following a mountpoint
and a couple of mount --move. The thing is, we don't need to make that
assumption at all - after the end of loop in follow_manage() we can check
if path->mnt has ended up unchanged and do mntput() if needed.
The BUG can be reproduced with the following test program:
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <sys/wait.h>
int main(int argc, char **argv)
{
int pid, ws;
struct stat buf;
pid = fork();
stat(argv[1], &buf);
if (pid > 0) wait(&ws);
return 0;
}
and the following procedure:
(1) Mount an NFS volume that on the server has something else mounted on a
subdirectory. For instance, I can mount / from my server:
mount warthog:/ /mnt -t nfs4 -r
On the server /data has another filesystem mounted on it, so NFS will see
a change in FSID as it walks down the path, and will mark /mnt/data as
being a mountpoint. This will cause the automount code to be triggered.
!!! Do not look inside the mounted fs at this point !!!
(2) Run the above program on a file within the submount to generate two
simultaneous automount requests:
/tmp/forkstat /mnt/data/testfile
(3) Unmount the automounted submount:
umount /mnt/data
(4) Unmount the original mount:
umount /mnt
At this point the kernel should throw a BUG with something like the
following:
BUG: Dentry ffff880032e3c5c0{i=2,n=} still in use (1) [unmount of nfs4 0:12]
Note that the bug appears on the root dentry of the original mount, not the
mountpoint and not the submount because sys_umount() hasn't got to its final
mntput_no_expire() yet, but this isn't so obvious from the call trace:
[<ffffffff8117cd82>] shrink_dcache_for_umount+0x69/0x82
[<ffffffff8116160e>] generic_shutdown_super+0x37/0x15b
[<ffffffffa00fae56>] ? nfs_super_return_all_delegations+0x2e/0x1b1 [nfs]
[<ffffffff811617f3>] kill_anon_super+0x1d/0x7e
[<ffffffffa00d0be1>] nfs4_kill_super+0x60/0xb6 [nfs]
[<ffffffff81161c17>] deactivate_locked_super+0x34/0x83
[<ffffffff811629ff>] deactivate_super+0x6f/0x7b
[<ffffffff81186261>] mntput_no_expire+0x18d/0x199
[<ffffffff811862a8>] mntput+0x3b/0x44
[<ffffffff81186d87>] release_mounts+0xa2/0xbf
[<ffffffff811876af>] sys_umount+0x47a/0x4ba
[<ffffffff8109e1ca>] ? trace_hardirqs_on_caller+0x1fd/0x22f
[<ffffffff816ea86b>] system_call_fastpath+0x16/0x1b
as do_umount() is inlined. However, you can see release_mounts() in there.
Note also that it may be necessary to have multiple CPU cores to be able to
trigger this bug.
Tested-by: Jeff Layton <jlayton@redhat.com>
Tested-by: Ian Kent <raven@themaw.net>
Signed-off-by: David Howells <dhowells@redhat.com>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-06-16 21:10:06 +07:00
|
|
|
|
|
|
|
if (need_mntput && path->mnt == mnt)
|
|
|
|
mntput(path->mnt);
|
|
|
|
if (ret == -EISDIR)
|
|
|
|
ret = 0;
|
2011-11-08 04:21:26 +07:00
|
|
|
return ret < 0 ? ret : need_mntput;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
Add a dentry op to allow processes to be held during pathwalk transit
Add a dentry op (d_manage) to permit a filesystem to hold a process and make it
sleep when it tries to transit away from one of that filesystem's directories
during a pathwalk. The operation is keyed off a new dentry flag
(DCACHE_MANAGE_TRANSIT).
The filesystem is allowed to be selective about which processes it holds and
which it permits to continue on or prohibits from transiting from each flagged
directory. This will allow autofs to hold up client processes whilst letting
its userspace daemon through to maintain the directory or the stuff behind it
or mounted upon it.
The ->d_manage() dentry operation:
int (*d_manage)(struct path *path, bool mounting_here);
takes a pointer to the directory about to be transited away from and a flag
indicating whether the transit is undertaken by do_add_mount() or
do_move_mount() skipping through a pile of filesystems mounted on a mountpoint.
It should return 0 if successful and to let the process continue on its way;
-EISDIR to prohibit the caller from skipping to overmounted filesystems or
automounting, and to use this directory; or some other error code to return to
the user.
->d_manage() is called with namespace_sem writelocked if mounting_here is true
and no other locks held, so it may sleep. However, if mounting_here is true,
it may not initiate or wait for a mount or unmount upon the parameter
directory, even if the act is actually performed by userspace.
Within fs/namei.c, follow_managed() is extended to check with d_manage() first
on each managed directory, before transiting away from it or attempting to
automount upon it.
follow_down() is renamed follow_down_one() and should only be used where the
filesystem deliberately intends to avoid management steps (e.g. autofs).
A new follow_down() is added that incorporates the loop done by all other
callers of follow_down() (do_add/move_mount(), autofs and NFSD; whilst AFS, NFS
and CIFS do use it, their use is removed by converting them to use
d_automount()). The new follow_down() calls d_manage() as appropriate. It
also takes an extra parameter to indicate if it is being called from mount code
(with namespace_sem writelocked) which it passes to d_manage(). follow_down()
ignores automount points so that it can be used to mount on them.
__follow_mount_rcu() is made to abort rcu-walk mode if it hits a directory with
DCACHE_MANAGE_TRANSIT set on the basis that we're probably going to have to
sleep. It would be possible to enter d_manage() in rcu-walk mode too, and have
that determine whether to abort or not itself. That would allow the autofs
daemon to continue on in rcu-walk mode.
Note that DCACHE_MANAGE_TRANSIT on a directory should be cleared when it isn't
required as every tranist from that directory will cause d_manage() to be
invoked. It can always be set again when necessary.
==========================
WHAT THIS MEANS FOR AUTOFS
==========================
Autofs currently uses the lookup() inode op and the d_revalidate() dentry op to
trigger the automounting of indirect mounts, and both of these can be called
with i_mutex held.
autofs knows that the i_mutex will be held by the caller in lookup(), and so
can drop it before invoking the daemon - but this isn't so for d_revalidate(),
since the lock is only held on _some_ of the code paths that call it. This
means that autofs can't risk dropping i_mutex from its d_revalidate() function
before it calls the daemon.
The bug could manifest itself as, for example, a process that's trying to
validate an automount dentry that gets made to wait because that dentry is
expired and needs cleaning up:
mkdir S ffffffff8014e05a 0 32580 24956
Call Trace:
[<ffffffff885371fd>] :autofs4:autofs4_wait+0x674/0x897
[<ffffffff80127f7d>] avc_has_perm+0x46/0x58
[<ffffffff8009fdcf>] autoremove_wake_function+0x0/0x2e
[<ffffffff88537be6>] :autofs4:autofs4_expire_wait+0x41/0x6b
[<ffffffff88535cfc>] :autofs4:autofs4_revalidate+0x91/0x149
[<ffffffff80036d96>] __lookup_hash+0xa0/0x12f
[<ffffffff80057a2f>] lookup_create+0x46/0x80
[<ffffffff800e6e31>] sys_mkdirat+0x56/0xe4
versus the automount daemon which wants to remove that dentry, but can't
because the normal process is holding the i_mutex lock:
automount D ffffffff8014e05a 0 32581 1 32561
Call Trace:
[<ffffffff80063c3f>] __mutex_lock_slowpath+0x60/0x9b
[<ffffffff8000ccf1>] do_path_lookup+0x2ca/0x2f1
[<ffffffff80063c89>] .text.lock.mutex+0xf/0x14
[<ffffffff800e6d55>] do_rmdir+0x77/0xde
[<ffffffff8005d229>] tracesys+0x71/0xe0
[<ffffffff8005d28d>] tracesys+0xd5/0xe0
which means that the system is deadlocked.
This patch allows autofs to hold up normal processes whilst the daemon goes
ahead and does things to the dentry tree behind the automouter point without
risking a deadlock as almost no locks are held in d_manage() and none in
d_automount().
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:26 +07:00
|
|
|
int follow_down_one(struct path *path)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
struct vfsmount *mounted;
|
|
|
|
|
2009-04-19 01:06:57 +07:00
|
|
|
mounted = lookup_mnt(path);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (mounted) {
|
2009-04-19 00:58:15 +07:00
|
|
|
dput(path->dentry);
|
|
|
|
mntput(path->mnt);
|
|
|
|
path->mnt = mounted;
|
|
|
|
path->dentry = dget(mounted->mnt_root);
|
2005-04-17 05:20:36 +07:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2011-03-25 00:51:02 +07:00
|
|
|
static inline bool managed_dentry_might_block(struct dentry *dentry)
|
|
|
|
{
|
|
|
|
return (dentry->d_flags & DCACHE_MANAGE_TRANSIT &&
|
|
|
|
dentry->d_op->d_manage(dentry, true) < 0);
|
|
|
|
}
|
|
|
|
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
/*
|
2011-05-27 17:50:06 +07:00
|
|
|
* Try to skip to top of mountpoint pile in rcuwalk mode. Fail if
|
|
|
|
* we meet a managed dentry that would need blocking.
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
*/
|
|
|
|
static bool __follow_mount_rcu(struct nameidata *nd, struct path *path,
|
2011-05-27 17:50:06 +07:00
|
|
|
struct inode **inode)
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
{
|
2011-03-25 00:51:02 +07:00
|
|
|
for (;;) {
|
2011-11-25 06:22:03 +07:00
|
|
|
struct mount *mounted;
|
2011-03-25 00:51:02 +07:00
|
|
|
/*
|
|
|
|
* Don't forget we might have a non-mountpoint managed dentry
|
|
|
|
* that wants to block transit.
|
|
|
|
*/
|
2011-05-27 17:50:06 +07:00
|
|
|
if (unlikely(managed_dentry_might_block(path->dentry)))
|
2011-01-15 01:46:51 +07:00
|
|
|
return false;
|
2011-03-25 00:51:02 +07:00
|
|
|
|
|
|
|
if (!d_mountpoint(path->dentry))
|
|
|
|
break;
|
|
|
|
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
mounted = __lookup_mnt(path->mnt, path->dentry, 1);
|
|
|
|
if (!mounted)
|
|
|
|
break;
|
2011-11-25 06:22:03 +07:00
|
|
|
path->mnt = &mounted->mnt;
|
|
|
|
path->dentry = mounted->mnt.mnt_root;
|
2011-11-08 04:21:26 +07:00
|
|
|
nd->flags |= LOOKUP_JUMPED;
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
nd->seq = read_seqcount_begin(&path->dentry->d_seq);
|
2011-07-19 05:43:29 +07:00
|
|
|
/*
|
|
|
|
* Update the inode too. We don't need to re-check the
|
|
|
|
* dentry sequence number here after this d_inode read,
|
|
|
|
* because a mount-point is always pinned.
|
|
|
|
*/
|
|
|
|
*inode = path->dentry->d_inode;
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
}
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2011-05-27 17:53:39 +07:00
|
|
|
static void follow_mount_rcu(struct nameidata *nd)
|
2011-05-27 17:50:06 +07:00
|
|
|
{
|
2011-05-27 17:53:39 +07:00
|
|
|
while (d_mountpoint(nd->path.dentry)) {
|
2011-11-25 06:22:03 +07:00
|
|
|
struct mount *mounted;
|
2011-05-27 17:53:39 +07:00
|
|
|
mounted = __lookup_mnt(nd->path.mnt, nd->path.dentry, 1);
|
2011-05-27 17:50:06 +07:00
|
|
|
if (!mounted)
|
|
|
|
break;
|
2011-11-25 06:22:03 +07:00
|
|
|
nd->path.mnt = &mounted->mnt;
|
|
|
|
nd->path.dentry = mounted->mnt.mnt_root;
|
2011-05-27 17:53:39 +07:00
|
|
|
nd->seq = read_seqcount_begin(&nd->path.dentry->d_seq);
|
2011-05-27 17:50:06 +07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
static int follow_dotdot_rcu(struct nameidata *nd)
|
|
|
|
{
|
|
|
|
set_root_rcu(nd);
|
|
|
|
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
while (1) {
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
if (nd->path.dentry == nd->root.dentry &&
|
|
|
|
nd->path.mnt == nd->root.mnt) {
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
if (nd->path.dentry != nd->path.mnt->mnt_root) {
|
|
|
|
struct dentry *old = nd->path.dentry;
|
|
|
|
struct dentry *parent = old->d_parent;
|
|
|
|
unsigned seq;
|
|
|
|
|
|
|
|
seq = read_seqcount_begin(&parent->d_seq);
|
|
|
|
if (read_seqcount_retry(&old->d_seq, nd->seq))
|
2011-03-05 02:35:59 +07:00
|
|
|
goto failed;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
nd->path.dentry = parent;
|
|
|
|
nd->seq = seq;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
if (!follow_up_rcu(&nd->path))
|
|
|
|
break;
|
|
|
|
nd->seq = read_seqcount_begin(&nd->path.dentry->d_seq);
|
|
|
|
}
|
2011-05-27 17:53:39 +07:00
|
|
|
follow_mount_rcu(nd);
|
|
|
|
nd->inode = nd->path.dentry->d_inode;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
return 0;
|
2011-03-05 02:35:59 +07:00
|
|
|
|
|
|
|
failed:
|
|
|
|
nd->flags &= ~LOOKUP_RCU;
|
2011-03-10 11:04:47 +07:00
|
|
|
if (!(nd->flags & LOOKUP_ROOT))
|
|
|
|
nd->root.mnt = NULL;
|
2011-03-05 02:35:59 +07:00
|
|
|
rcu_read_unlock();
|
2012-05-08 11:02:02 +07:00
|
|
|
br_read_unlock(&vfsmount_lock);
|
2011-03-05 02:35:59 +07:00
|
|
|
return -ECHILD;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
}
|
|
|
|
|
Add a dentry op to allow processes to be held during pathwalk transit
Add a dentry op (d_manage) to permit a filesystem to hold a process and make it
sleep when it tries to transit away from one of that filesystem's directories
during a pathwalk. The operation is keyed off a new dentry flag
(DCACHE_MANAGE_TRANSIT).
The filesystem is allowed to be selective about which processes it holds and
which it permits to continue on or prohibits from transiting from each flagged
directory. This will allow autofs to hold up client processes whilst letting
its userspace daemon through to maintain the directory or the stuff behind it
or mounted upon it.
The ->d_manage() dentry operation:
int (*d_manage)(struct path *path, bool mounting_here);
takes a pointer to the directory about to be transited away from and a flag
indicating whether the transit is undertaken by do_add_mount() or
do_move_mount() skipping through a pile of filesystems mounted on a mountpoint.
It should return 0 if successful and to let the process continue on its way;
-EISDIR to prohibit the caller from skipping to overmounted filesystems or
automounting, and to use this directory; or some other error code to return to
the user.
->d_manage() is called with namespace_sem writelocked if mounting_here is true
and no other locks held, so it may sleep. However, if mounting_here is true,
it may not initiate or wait for a mount or unmount upon the parameter
directory, even if the act is actually performed by userspace.
Within fs/namei.c, follow_managed() is extended to check with d_manage() first
on each managed directory, before transiting away from it or attempting to
automount upon it.
follow_down() is renamed follow_down_one() and should only be used where the
filesystem deliberately intends to avoid management steps (e.g. autofs).
A new follow_down() is added that incorporates the loop done by all other
callers of follow_down() (do_add/move_mount(), autofs and NFSD; whilst AFS, NFS
and CIFS do use it, their use is removed by converting them to use
d_automount()). The new follow_down() calls d_manage() as appropriate. It
also takes an extra parameter to indicate if it is being called from mount code
(with namespace_sem writelocked) which it passes to d_manage(). follow_down()
ignores automount points so that it can be used to mount on them.
__follow_mount_rcu() is made to abort rcu-walk mode if it hits a directory with
DCACHE_MANAGE_TRANSIT set on the basis that we're probably going to have to
sleep. It would be possible to enter d_manage() in rcu-walk mode too, and have
that determine whether to abort or not itself. That would allow the autofs
daemon to continue on in rcu-walk mode.
Note that DCACHE_MANAGE_TRANSIT on a directory should be cleared when it isn't
required as every tranist from that directory will cause d_manage() to be
invoked. It can always be set again when necessary.
==========================
WHAT THIS MEANS FOR AUTOFS
==========================
Autofs currently uses the lookup() inode op and the d_revalidate() dentry op to
trigger the automounting of indirect mounts, and both of these can be called
with i_mutex held.
autofs knows that the i_mutex will be held by the caller in lookup(), and so
can drop it before invoking the daemon - but this isn't so for d_revalidate(),
since the lock is only held on _some_ of the code paths that call it. This
means that autofs can't risk dropping i_mutex from its d_revalidate() function
before it calls the daemon.
The bug could manifest itself as, for example, a process that's trying to
validate an automount dentry that gets made to wait because that dentry is
expired and needs cleaning up:
mkdir S ffffffff8014e05a 0 32580 24956
Call Trace:
[<ffffffff885371fd>] :autofs4:autofs4_wait+0x674/0x897
[<ffffffff80127f7d>] avc_has_perm+0x46/0x58
[<ffffffff8009fdcf>] autoremove_wake_function+0x0/0x2e
[<ffffffff88537be6>] :autofs4:autofs4_expire_wait+0x41/0x6b
[<ffffffff88535cfc>] :autofs4:autofs4_revalidate+0x91/0x149
[<ffffffff80036d96>] __lookup_hash+0xa0/0x12f
[<ffffffff80057a2f>] lookup_create+0x46/0x80
[<ffffffff800e6e31>] sys_mkdirat+0x56/0xe4
versus the automount daemon which wants to remove that dentry, but can't
because the normal process is holding the i_mutex lock:
automount D ffffffff8014e05a 0 32581 1 32561
Call Trace:
[<ffffffff80063c3f>] __mutex_lock_slowpath+0x60/0x9b
[<ffffffff8000ccf1>] do_path_lookup+0x2ca/0x2f1
[<ffffffff80063c89>] .text.lock.mutex+0xf/0x14
[<ffffffff800e6d55>] do_rmdir+0x77/0xde
[<ffffffff8005d229>] tracesys+0x71/0xe0
[<ffffffff8005d28d>] tracesys+0xd5/0xe0
which means that the system is deadlocked.
This patch allows autofs to hold up normal processes whilst the daemon goes
ahead and does things to the dentry tree behind the automouter point without
risking a deadlock as almost no locks are held in d_manage() and none in
d_automount().
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:26 +07:00
|
|
|
/*
|
|
|
|
* Follow down to the covering mount currently visible to userspace. At each
|
|
|
|
* point, the filesystem owning that dentry may be queried as to whether the
|
|
|
|
* caller is permitted to proceed or not.
|
|
|
|
*/
|
2011-03-18 20:04:20 +07:00
|
|
|
int follow_down(struct path *path)
|
Add a dentry op to allow processes to be held during pathwalk transit
Add a dentry op (d_manage) to permit a filesystem to hold a process and make it
sleep when it tries to transit away from one of that filesystem's directories
during a pathwalk. The operation is keyed off a new dentry flag
(DCACHE_MANAGE_TRANSIT).
The filesystem is allowed to be selective about which processes it holds and
which it permits to continue on or prohibits from transiting from each flagged
directory. This will allow autofs to hold up client processes whilst letting
its userspace daemon through to maintain the directory or the stuff behind it
or mounted upon it.
The ->d_manage() dentry operation:
int (*d_manage)(struct path *path, bool mounting_here);
takes a pointer to the directory about to be transited away from and a flag
indicating whether the transit is undertaken by do_add_mount() or
do_move_mount() skipping through a pile of filesystems mounted on a mountpoint.
It should return 0 if successful and to let the process continue on its way;
-EISDIR to prohibit the caller from skipping to overmounted filesystems or
automounting, and to use this directory; or some other error code to return to
the user.
->d_manage() is called with namespace_sem writelocked if mounting_here is true
and no other locks held, so it may sleep. However, if mounting_here is true,
it may not initiate or wait for a mount or unmount upon the parameter
directory, even if the act is actually performed by userspace.
Within fs/namei.c, follow_managed() is extended to check with d_manage() first
on each managed directory, before transiting away from it or attempting to
automount upon it.
follow_down() is renamed follow_down_one() and should only be used where the
filesystem deliberately intends to avoid management steps (e.g. autofs).
A new follow_down() is added that incorporates the loop done by all other
callers of follow_down() (do_add/move_mount(), autofs and NFSD; whilst AFS, NFS
and CIFS do use it, their use is removed by converting them to use
d_automount()). The new follow_down() calls d_manage() as appropriate. It
also takes an extra parameter to indicate if it is being called from mount code
(with namespace_sem writelocked) which it passes to d_manage(). follow_down()
ignores automount points so that it can be used to mount on them.
__follow_mount_rcu() is made to abort rcu-walk mode if it hits a directory with
DCACHE_MANAGE_TRANSIT set on the basis that we're probably going to have to
sleep. It would be possible to enter d_manage() in rcu-walk mode too, and have
that determine whether to abort or not itself. That would allow the autofs
daemon to continue on in rcu-walk mode.
Note that DCACHE_MANAGE_TRANSIT on a directory should be cleared when it isn't
required as every tranist from that directory will cause d_manage() to be
invoked. It can always be set again when necessary.
==========================
WHAT THIS MEANS FOR AUTOFS
==========================
Autofs currently uses the lookup() inode op and the d_revalidate() dentry op to
trigger the automounting of indirect mounts, and both of these can be called
with i_mutex held.
autofs knows that the i_mutex will be held by the caller in lookup(), and so
can drop it before invoking the daemon - but this isn't so for d_revalidate(),
since the lock is only held on _some_ of the code paths that call it. This
means that autofs can't risk dropping i_mutex from its d_revalidate() function
before it calls the daemon.
The bug could manifest itself as, for example, a process that's trying to
validate an automount dentry that gets made to wait because that dentry is
expired and needs cleaning up:
mkdir S ffffffff8014e05a 0 32580 24956
Call Trace:
[<ffffffff885371fd>] :autofs4:autofs4_wait+0x674/0x897
[<ffffffff80127f7d>] avc_has_perm+0x46/0x58
[<ffffffff8009fdcf>] autoremove_wake_function+0x0/0x2e
[<ffffffff88537be6>] :autofs4:autofs4_expire_wait+0x41/0x6b
[<ffffffff88535cfc>] :autofs4:autofs4_revalidate+0x91/0x149
[<ffffffff80036d96>] __lookup_hash+0xa0/0x12f
[<ffffffff80057a2f>] lookup_create+0x46/0x80
[<ffffffff800e6e31>] sys_mkdirat+0x56/0xe4
versus the automount daemon which wants to remove that dentry, but can't
because the normal process is holding the i_mutex lock:
automount D ffffffff8014e05a 0 32581 1 32561
Call Trace:
[<ffffffff80063c3f>] __mutex_lock_slowpath+0x60/0x9b
[<ffffffff8000ccf1>] do_path_lookup+0x2ca/0x2f1
[<ffffffff80063c89>] .text.lock.mutex+0xf/0x14
[<ffffffff800e6d55>] do_rmdir+0x77/0xde
[<ffffffff8005d229>] tracesys+0x71/0xe0
[<ffffffff8005d28d>] tracesys+0xd5/0xe0
which means that the system is deadlocked.
This patch allows autofs to hold up normal processes whilst the daemon goes
ahead and does things to the dentry tree behind the automouter point without
risking a deadlock as almost no locks are held in d_manage() and none in
d_automount().
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:26 +07:00
|
|
|
{
|
|
|
|
unsigned managed;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
while (managed = ACCESS_ONCE(path->dentry->d_flags),
|
|
|
|
unlikely(managed & DCACHE_MANAGED_DENTRY)) {
|
|
|
|
/* Allow the filesystem to manage the transit without i_mutex
|
|
|
|
* being held.
|
|
|
|
*
|
|
|
|
* We indicate to the filesystem if someone is trying to mount
|
|
|
|
* something here. This gives autofs the chance to deny anyone
|
|
|
|
* other than its daemon the right to mount on its
|
|
|
|
* superstructure.
|
|
|
|
*
|
|
|
|
* The filesystem may sleep at this point.
|
|
|
|
*/
|
|
|
|
if (managed & DCACHE_MANAGE_TRANSIT) {
|
|
|
|
BUG_ON(!path->dentry->d_op);
|
|
|
|
BUG_ON(!path->dentry->d_op->d_manage);
|
2011-01-15 01:46:51 +07:00
|
|
|
ret = path->dentry->d_op->d_manage(
|
2011-03-18 20:09:02 +07:00
|
|
|
path->dentry, false);
|
Add a dentry op to allow processes to be held during pathwalk transit
Add a dentry op (d_manage) to permit a filesystem to hold a process and make it
sleep when it tries to transit away from one of that filesystem's directories
during a pathwalk. The operation is keyed off a new dentry flag
(DCACHE_MANAGE_TRANSIT).
The filesystem is allowed to be selective about which processes it holds and
which it permits to continue on or prohibits from transiting from each flagged
directory. This will allow autofs to hold up client processes whilst letting
its userspace daemon through to maintain the directory or the stuff behind it
or mounted upon it.
The ->d_manage() dentry operation:
int (*d_manage)(struct path *path, bool mounting_here);
takes a pointer to the directory about to be transited away from and a flag
indicating whether the transit is undertaken by do_add_mount() or
do_move_mount() skipping through a pile of filesystems mounted on a mountpoint.
It should return 0 if successful and to let the process continue on its way;
-EISDIR to prohibit the caller from skipping to overmounted filesystems or
automounting, and to use this directory; or some other error code to return to
the user.
->d_manage() is called with namespace_sem writelocked if mounting_here is true
and no other locks held, so it may sleep. However, if mounting_here is true,
it may not initiate or wait for a mount or unmount upon the parameter
directory, even if the act is actually performed by userspace.
Within fs/namei.c, follow_managed() is extended to check with d_manage() first
on each managed directory, before transiting away from it or attempting to
automount upon it.
follow_down() is renamed follow_down_one() and should only be used where the
filesystem deliberately intends to avoid management steps (e.g. autofs).
A new follow_down() is added that incorporates the loop done by all other
callers of follow_down() (do_add/move_mount(), autofs and NFSD; whilst AFS, NFS
and CIFS do use it, their use is removed by converting them to use
d_automount()). The new follow_down() calls d_manage() as appropriate. It
also takes an extra parameter to indicate if it is being called from mount code
(with namespace_sem writelocked) which it passes to d_manage(). follow_down()
ignores automount points so that it can be used to mount on them.
__follow_mount_rcu() is made to abort rcu-walk mode if it hits a directory with
DCACHE_MANAGE_TRANSIT set on the basis that we're probably going to have to
sleep. It would be possible to enter d_manage() in rcu-walk mode too, and have
that determine whether to abort or not itself. That would allow the autofs
daemon to continue on in rcu-walk mode.
Note that DCACHE_MANAGE_TRANSIT on a directory should be cleared when it isn't
required as every tranist from that directory will cause d_manage() to be
invoked. It can always be set again when necessary.
==========================
WHAT THIS MEANS FOR AUTOFS
==========================
Autofs currently uses the lookup() inode op and the d_revalidate() dentry op to
trigger the automounting of indirect mounts, and both of these can be called
with i_mutex held.
autofs knows that the i_mutex will be held by the caller in lookup(), and so
can drop it before invoking the daemon - but this isn't so for d_revalidate(),
since the lock is only held on _some_ of the code paths that call it. This
means that autofs can't risk dropping i_mutex from its d_revalidate() function
before it calls the daemon.
The bug could manifest itself as, for example, a process that's trying to
validate an automount dentry that gets made to wait because that dentry is
expired and needs cleaning up:
mkdir S ffffffff8014e05a 0 32580 24956
Call Trace:
[<ffffffff885371fd>] :autofs4:autofs4_wait+0x674/0x897
[<ffffffff80127f7d>] avc_has_perm+0x46/0x58
[<ffffffff8009fdcf>] autoremove_wake_function+0x0/0x2e
[<ffffffff88537be6>] :autofs4:autofs4_expire_wait+0x41/0x6b
[<ffffffff88535cfc>] :autofs4:autofs4_revalidate+0x91/0x149
[<ffffffff80036d96>] __lookup_hash+0xa0/0x12f
[<ffffffff80057a2f>] lookup_create+0x46/0x80
[<ffffffff800e6e31>] sys_mkdirat+0x56/0xe4
versus the automount daemon which wants to remove that dentry, but can't
because the normal process is holding the i_mutex lock:
automount D ffffffff8014e05a 0 32581 1 32561
Call Trace:
[<ffffffff80063c3f>] __mutex_lock_slowpath+0x60/0x9b
[<ffffffff8000ccf1>] do_path_lookup+0x2ca/0x2f1
[<ffffffff80063c89>] .text.lock.mutex+0xf/0x14
[<ffffffff800e6d55>] do_rmdir+0x77/0xde
[<ffffffff8005d229>] tracesys+0x71/0xe0
[<ffffffff8005d28d>] tracesys+0xd5/0xe0
which means that the system is deadlocked.
This patch allows autofs to hold up normal processes whilst the daemon goes
ahead and does things to the dentry tree behind the automouter point without
risking a deadlock as almost no locks are held in d_manage() and none in
d_automount().
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:26 +07:00
|
|
|
if (ret < 0)
|
|
|
|
return ret == -EISDIR ? 0 : ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Transit to a mounted filesystem. */
|
|
|
|
if (managed & DCACHE_MOUNTED) {
|
|
|
|
struct vfsmount *mounted = lookup_mnt(path);
|
|
|
|
if (!mounted)
|
|
|
|
break;
|
|
|
|
dput(path->dentry);
|
|
|
|
mntput(path->mnt);
|
|
|
|
path->mnt = mounted;
|
|
|
|
path->dentry = dget(mounted->mnt_root);
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Don't handle automount points here */
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
/*
|
|
|
|
* Skip to top of mountpoint pile in refwalk mode for follow_dotdot()
|
|
|
|
*/
|
|
|
|
static void follow_mount(struct path *path)
|
|
|
|
{
|
|
|
|
while (d_mountpoint(path->dentry)) {
|
|
|
|
struct vfsmount *mounted = lookup_mnt(path);
|
|
|
|
if (!mounted)
|
|
|
|
break;
|
|
|
|
dput(path->dentry);
|
|
|
|
mntput(path->mnt);
|
|
|
|
path->mnt = mounted;
|
|
|
|
path->dentry = dget(mounted->mnt_root);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
static void follow_dotdot(struct nameidata *nd)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2009-04-07 22:49:53 +07:00
|
|
|
set_root(nd);
|
2006-09-29 16:01:22 +07:00
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
while(1) {
|
2008-02-15 10:34:32 +07:00
|
|
|
struct dentry *old = nd->path.dentry;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2009-04-07 22:49:53 +07:00
|
|
|
if (nd->path.dentry == nd->root.dentry &&
|
|
|
|
nd->path.mnt == nd->root.mnt) {
|
2005-04-17 05:20:36 +07:00
|
|
|
break;
|
|
|
|
}
|
2008-02-15 10:34:32 +07:00
|
|
|
if (nd->path.dentry != nd->path.mnt->mnt_root) {
|
2010-01-31 03:47:29 +07:00
|
|
|
/* rare case of legitimate dget_parent()... */
|
|
|
|
nd->path.dentry = dget_parent(nd->path.dentry);
|
2005-04-17 05:20:36 +07:00
|
|
|
dput(old);
|
|
|
|
break;
|
|
|
|
}
|
2010-01-31 03:47:29 +07:00
|
|
|
if (!follow_up(&nd->path))
|
2005-04-17 05:20:36 +07:00
|
|
|
break;
|
|
|
|
}
|
2009-04-19 00:59:41 +07:00
|
|
|
follow_mount(&nd->path);
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
nd->inode = nd->path.dentry->d_inode;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2010-08-18 01:37:31 +07:00
|
|
|
/*
|
2012-03-26 17:54:24 +07:00
|
|
|
* This looks up the name in dcache, possibly revalidates the old dentry and
|
|
|
|
* allocates a new one if not found or not valid. In the need_lookup argument
|
|
|
|
* returns whether i_op->lookup is necessary.
|
|
|
|
*
|
|
|
|
* dir->d_inode->i_mutex must be held
|
2010-08-18 01:37:31 +07:00
|
|
|
*/
|
2012-03-26 17:54:24 +07:00
|
|
|
static struct dentry *lookup_dcache(struct qstr *name, struct dentry *dir,
|
|
|
|
struct nameidata *nd, bool *need_lookup)
|
2010-08-18 01:37:31 +07:00
|
|
|
{
|
|
|
|
struct dentry *dentry;
|
2012-03-26 17:54:24 +07:00
|
|
|
int error;
|
2010-08-18 01:37:31 +07:00
|
|
|
|
2012-03-26 17:54:24 +07:00
|
|
|
*need_lookup = false;
|
|
|
|
dentry = d_lookup(dir, name);
|
|
|
|
if (dentry) {
|
|
|
|
if (d_need_lookup(dentry)) {
|
|
|
|
*need_lookup = true;
|
|
|
|
} else if (dentry->d_flags & DCACHE_OP_REVALIDATE) {
|
|
|
|
error = d_revalidate(dentry, nd);
|
|
|
|
if (unlikely(error <= 0)) {
|
|
|
|
if (error < 0) {
|
|
|
|
dput(dentry);
|
|
|
|
return ERR_PTR(error);
|
|
|
|
} else if (!d_invalidate(dentry)) {
|
|
|
|
dput(dentry);
|
|
|
|
dentry = NULL;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
2010-08-18 01:37:31 +07:00
|
|
|
|
2012-03-26 17:54:24 +07:00
|
|
|
if (!dentry) {
|
|
|
|
dentry = d_alloc(dir, name);
|
|
|
|
if (unlikely(!dentry))
|
|
|
|
return ERR_PTR(-ENOMEM);
|
2010-08-18 01:37:31 +07:00
|
|
|
|
2012-03-26 17:54:24 +07:00
|
|
|
*need_lookup = true;
|
2010-08-18 01:37:31 +07:00
|
|
|
}
|
|
|
|
return dentry;
|
|
|
|
}
|
|
|
|
|
2011-05-31 22:58:49 +07:00
|
|
|
/*
|
2012-03-26 17:54:24 +07:00
|
|
|
* Call i_op->lookup on the dentry. The dentry must be negative but may be
|
|
|
|
* hashed if it was pouplated with DCACHE_NEED_LOOKUP.
|
|
|
|
*
|
|
|
|
* dir->d_inode->i_mutex must be held
|
2011-05-31 22:58:49 +07:00
|
|
|
*/
|
2012-03-26 17:54:24 +07:00
|
|
|
static struct dentry *lookup_real(struct inode *dir, struct dentry *dentry,
|
|
|
|
struct nameidata *nd)
|
2011-05-31 22:58:49 +07:00
|
|
|
{
|
|
|
|
struct dentry *old;
|
|
|
|
|
|
|
|
/* Don't create child dentry for a dead directory. */
|
2012-03-26 17:54:24 +07:00
|
|
|
if (unlikely(IS_DEADDIR(dir))) {
|
2012-02-03 20:25:18 +07:00
|
|
|
dput(dentry);
|
2011-05-31 22:58:49 +07:00
|
|
|
return ERR_PTR(-ENOENT);
|
2012-02-03 20:25:18 +07:00
|
|
|
}
|
2011-05-31 22:58:49 +07:00
|
|
|
|
2012-03-26 17:54:24 +07:00
|
|
|
old = dir->i_op->lookup(dir, dentry, nd);
|
2011-05-31 22:58:49 +07:00
|
|
|
if (unlikely(old)) {
|
|
|
|
dput(dentry);
|
|
|
|
dentry = old;
|
|
|
|
}
|
|
|
|
return dentry;
|
|
|
|
}
|
|
|
|
|
2012-03-31 01:41:51 +07:00
|
|
|
static struct dentry *__lookup_hash(struct qstr *name,
|
|
|
|
struct dentry *base, struct nameidata *nd)
|
|
|
|
{
|
2012-03-26 17:54:24 +07:00
|
|
|
bool need_lookup;
|
2012-03-31 01:41:51 +07:00
|
|
|
struct dentry *dentry;
|
|
|
|
|
2012-03-26 17:54:24 +07:00
|
|
|
dentry = lookup_dcache(name, base, nd, &need_lookup);
|
|
|
|
if (!need_lookup)
|
|
|
|
return dentry;
|
2012-03-31 01:41:51 +07:00
|
|
|
|
2012-03-26 17:54:24 +07:00
|
|
|
return lookup_real(base->d_inode, dentry, nd);
|
2012-03-31 01:41:51 +07:00
|
|
|
}
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* It's more convoluted than I'd like it to be, but... it's still fairly
|
|
|
|
* small and for now I'd prefer to have fast path as straight as possible.
|
|
|
|
* It _is_ time-critical.
|
|
|
|
*/
|
2012-05-21 22:30:05 +07:00
|
|
|
static int lookup_fast(struct nameidata *nd, struct qstr *name,
|
|
|
|
struct path *path, struct inode **inode)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2008-02-15 10:34:32 +07:00
|
|
|
struct vfsmount *mnt = nd->path.mnt;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
struct dentry *dentry, *parent = nd->path.dentry;
|
untangle do_lookup()
That thing has devolved into rats nest of gotos; sane use of unlikely()
gets rid of that horror and gives much more readable structure:
* make a fast attempt to find a dentry; false negatives are OK.
In RCU mode if everything went fine, we are done, otherwise just drop
out of RCU. If we'd done (RCU) ->d_revalidate() and it had not refused
outright (i.e. didn't give us -ECHILD), remember its result.
* now we are not in RCU mode and hopefully have a dentry. If we
do not, lock parent, do full d_lookup() and if that has not found anything,
allocate and call ->lookup(). If we'd done that ->lookup(), remember that
dentry is good and we don't need to revalidate it.
* now we have a dentry. If it has ->d_revalidate() and we can't
skip it, call it.
* hopefully dentry is good; if not, either fail (in case of error)
or try to invalidate it. If d_invalidate() has succeeded, drop it and
retry everything as if original attempt had not found a dentry.
* now we can finish it up - deal with mountpoint crossing and
automount.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-03-11 16:44:53 +07:00
|
|
|
int need_reval = 1;
|
|
|
|
int status = 1;
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
int err;
|
|
|
|
|
fs: remove extra lookup in __lookup_hash
fs: remove extra lookup in __lookup_hash
Optimize lookup for create operations, where no dentry should often be
common-case. In cases where it is not, such as unlink, the added overhead
is much smaller than the removed.
Also, move comments about __d_lookup racyness to the __d_lookup call site.
d_lookup is intuitive; __d_lookup is what needs commenting. So in that same
vein, add kerneldoc comments to __d_lookup and clean up some of the comments:
- We are interested in how the RCU lookup works here, particularly with
renames. Make that explicit, and point to the document where it is explained
in more detail.
- RCU is pretty standard now, and macros make implementations pretty mindless.
If we want to know about RCU barrier details, we look in RCU code.
- Delete some boring legacy comments because we don't care much about how the
code used to work, more about the interesting parts of how it works now. So
comments about lazy LRU may be interesting, but would better be done in the
LRU or refcount management code.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2010-08-18 01:37:34 +07:00
|
|
|
/*
|
|
|
|
* Rename seqlock is not required here because in the off chance
|
|
|
|
* of a false negative due to a concurrent rename, we're going to
|
|
|
|
* do the non-racy lookup, below.
|
|
|
|
*/
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
if (nd->flags & LOOKUP_RCU) {
|
|
|
|
unsigned seq;
|
vfs: clean up __d_lookup_rcu() and dentry_cmp() interfaces
The calling conventions for __d_lookup_rcu() and dentry_cmp() are
annoying in different ways, and there is actually one single underlying
reason for both of the annoyances.
The fundamental reason is that we do the returned dentry sequence number
check inside __d_lookup_rcu() instead of doing it in the caller. This
results in two annoyances:
- __d_lookup_rcu() now not only needs to return the dentry and the
sequence number that goes along with the lookup, it also needs to
return the inode pointer that was validated by that sequence number
check.
- and because we did the sequence number check early (to validate the
name pointer and length) we also couldn't just pass the dentry itself
to dentry_cmp(), we had to pass the counted string that contained the
name.
So that sequence number decision caused two separate ugly calling
conventions.
Both of these problems would be solved if we just did the sequence
number check in the caller instead. There's only one caller, and that
caller already has to do the sequence number check for the parent
anyway, so just do that.
That allows us to stop returning the dentry->d_inode in that in-out
argument (pointer-to-pointer-to-inode), so we can make the inode
argument just a regular input inode pointer. The caller can just load
the inode from dentry->d_inode, and then do the sequence number check
after that to make sure that it's synchronized with the name we looked
up.
And it allows us to just pass in the dentry to dentry_cmp(), which is
what all the callers really wanted. Sure, dentry_cmp() has to be a bit
careful about the dentry (which is not stable during RCU lookup), but
that's actually very simple.
And now that dentry_cmp() can clearly see that the first string argument
is a dentry, we can use the direct word access for that, instead of the
careful unaligned zero-padding. The dentry name is always properly
aligned, since it is a single path component that is either embedded
into the dentry itself, or was allocated with kmalloc() (see __d_alloc).
Finally, this also uninlines the nasty slow-case for dentry comparisons:
that one *does* need to do a sequence number check, since it will call
in to the low-level filesystems, and we want to give those a stable
inode pointer and path component length/start arguments. Doing an extra
sequence check for that slow case is not a problem, though.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-05-05 04:59:14 +07:00
|
|
|
dentry = __d_lookup_rcu(parent, name, &seq, nd->inode);
|
untangle do_lookup()
That thing has devolved into rats nest of gotos; sane use of unlikely()
gets rid of that horror and gives much more readable structure:
* make a fast attempt to find a dentry; false negatives are OK.
In RCU mode if everything went fine, we are done, otherwise just drop
out of RCU. If we'd done (RCU) ->d_revalidate() and it had not refused
outright (i.e. didn't give us -ECHILD), remember its result.
* now we are not in RCU mode and hopefully have a dentry. If we
do not, lock parent, do full d_lookup() and if that has not found anything,
allocate and call ->lookup(). If we'd done that ->lookup(), remember that
dentry is good and we don't need to revalidate it.
* now we have a dentry. If it has ->d_revalidate() and we can't
skip it, call it.
* hopefully dentry is good; if not, either fail (in case of error)
or try to invalidate it. If d_invalidate() has succeeded, drop it and
retry everything as if original attempt had not found a dentry.
* now we can finish it up - deal with mountpoint crossing and
automount.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-03-11 16:44:53 +07:00
|
|
|
if (!dentry)
|
|
|
|
goto unlazy;
|
|
|
|
|
vfs: clean up __d_lookup_rcu() and dentry_cmp() interfaces
The calling conventions for __d_lookup_rcu() and dentry_cmp() are
annoying in different ways, and there is actually one single underlying
reason for both of the annoyances.
The fundamental reason is that we do the returned dentry sequence number
check inside __d_lookup_rcu() instead of doing it in the caller. This
results in two annoyances:
- __d_lookup_rcu() now not only needs to return the dentry and the
sequence number that goes along with the lookup, it also needs to
return the inode pointer that was validated by that sequence number
check.
- and because we did the sequence number check early (to validate the
name pointer and length) we also couldn't just pass the dentry itself
to dentry_cmp(), we had to pass the counted string that contained the
name.
So that sequence number decision caused two separate ugly calling
conventions.
Both of these problems would be solved if we just did the sequence
number check in the caller instead. There's only one caller, and that
caller already has to do the sequence number check for the parent
anyway, so just do that.
That allows us to stop returning the dentry->d_inode in that in-out
argument (pointer-to-pointer-to-inode), so we can make the inode
argument just a regular input inode pointer. The caller can just load
the inode from dentry->d_inode, and then do the sequence number check
after that to make sure that it's synchronized with the name we looked
up.
And it allows us to just pass in the dentry to dentry_cmp(), which is
what all the callers really wanted. Sure, dentry_cmp() has to be a bit
careful about the dentry (which is not stable during RCU lookup), but
that's actually very simple.
And now that dentry_cmp() can clearly see that the first string argument
is a dentry, we can use the direct word access for that, instead of the
careful unaligned zero-padding. The dentry name is always properly
aligned, since it is a single path component that is either embedded
into the dentry itself, or was allocated with kmalloc() (see __d_alloc).
Finally, this also uninlines the nasty slow-case for dentry comparisons:
that one *does* need to do a sequence number check, since it will call
in to the low-level filesystems, and we want to give those a stable
inode pointer and path component length/start arguments. Doing an extra
sequence check for that slow case is not a problem, though.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-05-05 04:59:14 +07:00
|
|
|
/*
|
|
|
|
* This sequence count validates that the inode matches
|
|
|
|
* the dentry name information from lookup.
|
|
|
|
*/
|
|
|
|
*inode = dentry->d_inode;
|
|
|
|
if (read_seqcount_retry(&dentry->d_seq, seq))
|
|
|
|
return -ECHILD;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This sequence count validates that the parent had no
|
|
|
|
* changes while we did the lookup of the dentry above.
|
|
|
|
*
|
|
|
|
* The memory barrier in read_seqcount_begin of child is
|
|
|
|
* enough, we can use __read_seqcount_retry here.
|
|
|
|
*/
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
if (__read_seqcount_retry(&parent->d_seq, nd->seq))
|
|
|
|
return -ECHILD;
|
|
|
|
nd->seq = seq;
|
untangle do_lookup()
That thing has devolved into rats nest of gotos; sane use of unlikely()
gets rid of that horror and gives much more readable structure:
* make a fast attempt to find a dentry; false negatives are OK.
In RCU mode if everything went fine, we are done, otherwise just drop
out of RCU. If we'd done (RCU) ->d_revalidate() and it had not refused
outright (i.e. didn't give us -ECHILD), remember its result.
* now we are not in RCU mode and hopefully have a dentry. If we
do not, lock parent, do full d_lookup() and if that has not found anything,
allocate and call ->lookup(). If we'd done that ->lookup(), remember that
dentry is good and we don't need to revalidate it.
* now we have a dentry. If it has ->d_revalidate() and we can't
skip it, call it.
* hopefully dentry is good; if not, either fail (in case of error)
or try to invalidate it. If d_invalidate() has succeeded, drop it and
retry everything as if original attempt had not found a dentry.
* now we can finish it up - deal with mountpoint crossing and
automount.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-03-11 16:44:53 +07:00
|
|
|
|
2012-03-26 17:54:19 +07:00
|
|
|
if (unlikely(d_need_lookup(dentry)))
|
|
|
|
goto unlazy;
|
2011-02-15 13:26:22 +07:00
|
|
|
if (unlikely(dentry->d_flags & DCACHE_OP_REVALIDATE)) {
|
untangle do_lookup()
That thing has devolved into rats nest of gotos; sane use of unlikely()
gets rid of that horror and gives much more readable structure:
* make a fast attempt to find a dentry; false negatives are OK.
In RCU mode if everything went fine, we are done, otherwise just drop
out of RCU. If we'd done (RCU) ->d_revalidate() and it had not refused
outright (i.e. didn't give us -ECHILD), remember its result.
* now we are not in RCU mode and hopefully have a dentry. If we
do not, lock parent, do full d_lookup() and if that has not found anything,
allocate and call ->lookup(). If we'd done that ->lookup(), remember that
dentry is good and we don't need to revalidate it.
* now we have a dentry. If it has ->d_revalidate() and we can't
skip it, call it.
* hopefully dentry is good; if not, either fail (in case of error)
or try to invalidate it. If d_invalidate() has succeeded, drop it and
retry everything as if original attempt had not found a dentry.
* now we can finish it up - deal with mountpoint crossing and
automount.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-03-11 16:44:53 +07:00
|
|
|
status = d_revalidate(dentry, nd);
|
|
|
|
if (unlikely(status <= 0)) {
|
|
|
|
if (status != -ECHILD)
|
|
|
|
need_reval = 0;
|
|
|
|
goto unlazy;
|
|
|
|
}
|
2011-02-15 13:26:22 +07:00
|
|
|
}
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
path->mnt = mnt;
|
|
|
|
path->dentry = dentry;
|
2011-05-27 18:03:15 +07:00
|
|
|
if (unlikely(!__follow_mount_rcu(nd, path, inode)))
|
|
|
|
goto unlazy;
|
|
|
|
if (unlikely(path->dentry->d_flags & DCACHE_NEED_AUTOMOUNT))
|
|
|
|
goto unlazy;
|
|
|
|
return 0;
|
untangle do_lookup()
That thing has devolved into rats nest of gotos; sane use of unlikely()
gets rid of that horror and gives much more readable structure:
* make a fast attempt to find a dentry; false negatives are OK.
In RCU mode if everything went fine, we are done, otherwise just drop
out of RCU. If we'd done (RCU) ->d_revalidate() and it had not refused
outright (i.e. didn't give us -ECHILD), remember its result.
* now we are not in RCU mode and hopefully have a dentry. If we
do not, lock parent, do full d_lookup() and if that has not found anything,
allocate and call ->lookup(). If we'd done that ->lookup(), remember that
dentry is good and we don't need to revalidate it.
* now we have a dentry. If it has ->d_revalidate() and we can't
skip it, call it.
* hopefully dentry is good; if not, either fail (in case of error)
or try to invalidate it. If d_invalidate() has succeeded, drop it and
retry everything as if original attempt had not found a dentry.
* now we can finish it up - deal with mountpoint crossing and
automount.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-03-11 16:44:53 +07:00
|
|
|
unlazy:
|
2011-03-25 21:32:48 +07:00
|
|
|
if (unlazy_walk(nd, dentry))
|
|
|
|
return -ECHILD;
|
untangle do_lookup()
That thing has devolved into rats nest of gotos; sane use of unlikely()
gets rid of that horror and gives much more readable structure:
* make a fast attempt to find a dentry; false negatives are OK.
In RCU mode if everything went fine, we are done, otherwise just drop
out of RCU. If we'd done (RCU) ->d_revalidate() and it had not refused
outright (i.e. didn't give us -ECHILD), remember its result.
* now we are not in RCU mode and hopefully have a dentry. If we
do not, lock parent, do full d_lookup() and if that has not found anything,
allocate and call ->lookup(). If we'd done that ->lookup(), remember that
dentry is good and we don't need to revalidate it.
* now we have a dentry. If it has ->d_revalidate() and we can't
skip it, call it.
* hopefully dentry is good; if not, either fail (in case of error)
or try to invalidate it. If d_invalidate() has succeeded, drop it and
retry everything as if original attempt had not found a dentry.
* now we can finish it up - deal with mountpoint crossing and
automount.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-03-11 16:44:53 +07:00
|
|
|
} else {
|
|
|
|
dentry = __d_lookup(parent, name);
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
}
|
untangle do_lookup()
That thing has devolved into rats nest of gotos; sane use of unlikely()
gets rid of that horror and gives much more readable structure:
* make a fast attempt to find a dentry; false negatives are OK.
In RCU mode if everything went fine, we are done, otherwise just drop
out of RCU. If we'd done (RCU) ->d_revalidate() and it had not refused
outright (i.e. didn't give us -ECHILD), remember its result.
* now we are not in RCU mode and hopefully have a dentry. If we
do not, lock parent, do full d_lookup() and if that has not found anything,
allocate and call ->lookup(). If we'd done that ->lookup(), remember that
dentry is good and we don't need to revalidate it.
* now we have a dentry. If it has ->d_revalidate() and we can't
skip it, call it.
* hopefully dentry is good; if not, either fail (in case of error)
or try to invalidate it. If d_invalidate() has succeeded, drop it and
retry everything as if original attempt had not found a dentry.
* now we can finish it up - deal with mountpoint crossing and
automount.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-03-11 16:44:53 +07:00
|
|
|
|
2012-03-31 01:48:04 +07:00
|
|
|
if (unlikely(!dentry))
|
|
|
|
goto need_lookup;
|
|
|
|
|
|
|
|
if (unlikely(d_need_lookup(dentry))) {
|
2011-05-31 22:58:49 +07:00
|
|
|
dput(dentry);
|
2012-03-31 01:48:04 +07:00
|
|
|
goto need_lookup;
|
untangle do_lookup()
That thing has devolved into rats nest of gotos; sane use of unlikely()
gets rid of that horror and gives much more readable structure:
* make a fast attempt to find a dentry; false negatives are OK.
In RCU mode if everything went fine, we are done, otherwise just drop
out of RCU. If we'd done (RCU) ->d_revalidate() and it had not refused
outright (i.e. didn't give us -ECHILD), remember its result.
* now we are not in RCU mode and hopefully have a dentry. If we
do not, lock parent, do full d_lookup() and if that has not found anything,
allocate and call ->lookup(). If we'd done that ->lookup(), remember that
dentry is good and we don't need to revalidate it.
* now we have a dentry. If it has ->d_revalidate() and we can't
skip it, call it.
* hopefully dentry is good; if not, either fail (in case of error)
or try to invalidate it. If d_invalidate() has succeeded, drop it and
retry everything as if original attempt had not found a dentry.
* now we can finish it up - deal with mountpoint crossing and
automount.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-03-11 16:44:53 +07:00
|
|
|
}
|
2012-03-31 01:48:04 +07:00
|
|
|
|
untangle do_lookup()
That thing has devolved into rats nest of gotos; sane use of unlikely()
gets rid of that horror and gives much more readable structure:
* make a fast attempt to find a dentry; false negatives are OK.
In RCU mode if everything went fine, we are done, otherwise just drop
out of RCU. If we'd done (RCU) ->d_revalidate() and it had not refused
outright (i.e. didn't give us -ECHILD), remember its result.
* now we are not in RCU mode and hopefully have a dentry. If we
do not, lock parent, do full d_lookup() and if that has not found anything,
allocate and call ->lookup(). If we'd done that ->lookup(), remember that
dentry is good and we don't need to revalidate it.
* now we have a dentry. If it has ->d_revalidate() and we can't
skip it, call it.
* hopefully dentry is good; if not, either fail (in case of error)
or try to invalidate it. If d_invalidate() has succeeded, drop it and
retry everything as if original attempt had not found a dentry.
* now we can finish it up - deal with mountpoint crossing and
automount.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-03-11 16:44:53 +07:00
|
|
|
if (unlikely(dentry->d_flags & DCACHE_OP_REVALIDATE) && need_reval)
|
|
|
|
status = d_revalidate(dentry, nd);
|
|
|
|
if (unlikely(status <= 0)) {
|
|
|
|
if (status < 0) {
|
|
|
|
dput(dentry);
|
|
|
|
return status;
|
|
|
|
}
|
|
|
|
if (!d_invalidate(dentry)) {
|
|
|
|
dput(dentry);
|
2012-03-31 01:48:04 +07:00
|
|
|
goto need_lookup;
|
untangle do_lookup()
That thing has devolved into rats nest of gotos; sane use of unlikely()
gets rid of that horror and gives much more readable structure:
* make a fast attempt to find a dentry; false negatives are OK.
In RCU mode if everything went fine, we are done, otherwise just drop
out of RCU. If we'd done (RCU) ->d_revalidate() and it had not refused
outright (i.e. didn't give us -ECHILD), remember its result.
* now we are not in RCU mode and hopefully have a dentry. If we
do not, lock parent, do full d_lookup() and if that has not found anything,
allocate and call ->lookup(). If we'd done that ->lookup(), remember that
dentry is good and we don't need to revalidate it.
* now we have a dentry. If it has ->d_revalidate() and we can't
skip it, call it.
* hopefully dentry is good; if not, either fail (in case of error)
or try to invalidate it. If d_invalidate() has succeeded, drop it and
retry everything as if original attempt had not found a dentry.
* now we can finish it up - deal with mountpoint crossing and
automount.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-03-11 16:44:53 +07:00
|
|
|
}
|
2011-02-15 13:26:22 +07:00
|
|
|
}
|
2012-05-21 22:30:05 +07:00
|
|
|
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
path->mnt = mnt;
|
|
|
|
path->dentry = dentry;
|
|
|
|
err = follow_managed(path, nd->flags);
|
2011-01-18 11:06:10 +07:00
|
|
|
if (unlikely(err < 0)) {
|
|
|
|
path_put_conditional(path, nd);
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
return err;
|
2011-01-18 11:06:10 +07:00
|
|
|
}
|
2011-11-08 04:21:26 +07:00
|
|
|
if (err)
|
|
|
|
nd->flags |= LOOKUP_JUMPED;
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
*inode = path->dentry->d_inode;
|
2005-04-17 05:20:36 +07:00
|
|
|
return 0;
|
2012-03-31 01:48:04 +07:00
|
|
|
|
|
|
|
need_lookup:
|
2012-05-21 22:30:05 +07:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Fast lookup failed, do it the slow way */
|
|
|
|
static int lookup_slow(struct nameidata *nd, struct qstr *name,
|
|
|
|
struct path *path)
|
|
|
|
{
|
|
|
|
struct dentry *dentry, *parent;
|
|
|
|
int err;
|
|
|
|
|
|
|
|
parent = nd->path.dentry;
|
2012-03-31 01:48:04 +07:00
|
|
|
BUG_ON(nd->inode != parent->d_inode);
|
|
|
|
|
|
|
|
mutex_lock(&parent->d_inode->i_mutex);
|
|
|
|
dentry = __lookup_hash(name, parent, nd);
|
|
|
|
mutex_unlock(&parent->d_inode->i_mutex);
|
|
|
|
if (IS_ERR(dentry))
|
|
|
|
return PTR_ERR(dentry);
|
2012-05-21 22:30:05 +07:00
|
|
|
path->mnt = nd->path.mnt;
|
|
|
|
path->dentry = dentry;
|
|
|
|
err = follow_managed(path, nd->flags);
|
|
|
|
if (unlikely(err < 0)) {
|
|
|
|
path_put_conditional(path, nd);
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
if (err)
|
|
|
|
nd->flags |= LOOKUP_JUMPED;
|
|
|
|
return 0;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2011-02-22 09:34:47 +07:00
|
|
|
static inline int may_lookup(struct nameidata *nd)
|
|
|
|
{
|
|
|
|
if (nd->flags & LOOKUP_RCU) {
|
2011-06-21 06:57:03 +07:00
|
|
|
int err = inode_permission(nd->inode, MAY_EXEC|MAY_NOT_BLOCK);
|
2011-02-22 09:34:47 +07:00
|
|
|
if (err != -ECHILD)
|
|
|
|
return err;
|
2011-03-25 21:32:48 +07:00
|
|
|
if (unlazy_walk(nd, NULL))
|
2011-02-22 09:34:47 +07:00
|
|
|
return -ECHILD;
|
|
|
|
}
|
2011-06-21 06:57:03 +07:00
|
|
|
return inode_permission(nd->inode, MAY_EXEC);
|
2011-02-22 09:34:47 +07:00
|
|
|
}
|
|
|
|
|
2011-03-05 02:22:06 +07:00
|
|
|
static inline int handle_dots(struct nameidata *nd, int type)
|
|
|
|
{
|
|
|
|
if (type == LAST_DOTDOT) {
|
|
|
|
if (nd->flags & LOOKUP_RCU) {
|
|
|
|
if (follow_dotdot_rcu(nd))
|
|
|
|
return -ECHILD;
|
|
|
|
} else
|
|
|
|
follow_dotdot(nd);
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2011-03-05 02:44:37 +07:00
|
|
|
static void terminate_walk(struct nameidata *nd)
|
|
|
|
{
|
|
|
|
if (!(nd->flags & LOOKUP_RCU)) {
|
|
|
|
path_put(&nd->path);
|
|
|
|
} else {
|
|
|
|
nd->flags &= ~LOOKUP_RCU;
|
2011-03-10 11:04:47 +07:00
|
|
|
if (!(nd->flags & LOOKUP_ROOT))
|
|
|
|
nd->root.mnt = NULL;
|
2011-03-05 02:44:37 +07:00
|
|
|
rcu_read_unlock();
|
2012-05-08 11:02:02 +07:00
|
|
|
br_read_unlock(&vfsmount_lock);
|
2011-03-05 02:44:37 +07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2011-08-07 12:45:50 +07:00
|
|
|
/*
|
|
|
|
* Do we need to follow links? We _really_ want to be able
|
|
|
|
* to do this check without having to look at inode->i_op,
|
|
|
|
* so we keep a cache of "no, this doesn't need follow_link"
|
|
|
|
* for the common case.
|
|
|
|
*/
|
2011-08-07 23:53:20 +07:00
|
|
|
static inline int should_follow_link(struct inode *inode, int follow)
|
2011-08-07 12:45:50 +07:00
|
|
|
{
|
|
|
|
if (unlikely(!(inode->i_opflags & IOP_NOFOLLOW))) {
|
|
|
|
if (likely(inode->i_op->follow_link))
|
|
|
|
return follow;
|
|
|
|
|
|
|
|
/* This gets set once for the inode lifetime */
|
|
|
|
spin_lock(&inode->i_lock);
|
|
|
|
inode->i_opflags |= IOP_NOFOLLOW;
|
|
|
|
spin_unlock(&inode->i_lock);
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2011-03-14 06:58:58 +07:00
|
|
|
static inline int walk_component(struct nameidata *nd, struct path *path,
|
|
|
|
struct qstr *name, int type, int follow)
|
|
|
|
{
|
|
|
|
struct inode *inode;
|
|
|
|
int err;
|
|
|
|
/*
|
|
|
|
* "." and ".." are special - ".." especially so because it has
|
|
|
|
* to be able to know about the current root directory and
|
|
|
|
* parent relationships.
|
|
|
|
*/
|
|
|
|
if (unlikely(type != LAST_NORM))
|
|
|
|
return handle_dots(nd, type);
|
2012-05-21 22:30:05 +07:00
|
|
|
err = lookup_fast(nd, name, path, &inode);
|
2011-03-14 06:58:58 +07:00
|
|
|
if (unlikely(err)) {
|
2012-05-21 22:30:05 +07:00
|
|
|
if (err < 0)
|
|
|
|
goto out_err;
|
|
|
|
|
|
|
|
err = lookup_slow(nd, name, path);
|
|
|
|
if (err < 0)
|
|
|
|
goto out_err;
|
|
|
|
|
|
|
|
inode = path->dentry->d_inode;
|
2011-03-14 06:58:58 +07:00
|
|
|
}
|
2012-05-21 22:30:05 +07:00
|
|
|
err = -ENOENT;
|
|
|
|
if (!inode)
|
|
|
|
goto out_path_put;
|
|
|
|
|
2011-08-07 23:53:20 +07:00
|
|
|
if (should_follow_link(inode, follow)) {
|
2011-03-25 21:32:48 +07:00
|
|
|
if (nd->flags & LOOKUP_RCU) {
|
|
|
|
if (unlikely(unlazy_walk(nd, path->dentry))) {
|
2012-05-21 22:30:05 +07:00
|
|
|
err = -ECHILD;
|
|
|
|
goto out_err;
|
2011-03-25 21:32:48 +07:00
|
|
|
}
|
|
|
|
}
|
2011-03-14 06:58:58 +07:00
|
|
|
BUG_ON(inode != path->dentry->d_inode);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
path_to_nameidata(path, nd);
|
|
|
|
nd->inode = inode;
|
|
|
|
return 0;
|
2012-05-21 22:30:05 +07:00
|
|
|
|
|
|
|
out_path_put:
|
|
|
|
path_to_nameidata(path, nd);
|
|
|
|
out_err:
|
|
|
|
terminate_walk(nd);
|
|
|
|
return err;
|
2011-03-14 06:58:58 +07:00
|
|
|
}
|
|
|
|
|
2011-03-15 08:54:55 +07:00
|
|
|
/*
|
|
|
|
* This limits recursive symlink follows to 8, while
|
|
|
|
* limiting consecutive symlinks to 40.
|
|
|
|
*
|
|
|
|
* Without that kind of total limit, nasty chains of consecutive
|
|
|
|
* symlinks can cause almost arbitrarily long lookups.
|
|
|
|
*/
|
|
|
|
static inline int nested_symlink(struct path *path, struct nameidata *nd)
|
|
|
|
{
|
|
|
|
int res;
|
|
|
|
|
|
|
|
if (unlikely(current->link_count >= MAX_NESTED_LINKS)) {
|
|
|
|
path_put_conditional(path, nd);
|
|
|
|
path_put(&nd->path);
|
|
|
|
return -ELOOP;
|
|
|
|
}
|
2011-05-21 12:19:59 +07:00
|
|
|
BUG_ON(nd->depth >= MAX_NESTED_LINKS);
|
2011-03-15 08:54:55 +07:00
|
|
|
|
|
|
|
nd->depth++;
|
|
|
|
current->link_count++;
|
|
|
|
|
|
|
|
do {
|
|
|
|
struct path link = *path;
|
|
|
|
void *cookie;
|
2011-03-15 09:20:34 +07:00
|
|
|
|
|
|
|
res = follow_link(&link, nd, &cookie);
|
2011-03-15 08:54:55 +07:00
|
|
|
if (!res)
|
|
|
|
res = walk_component(nd, path, &nd->last,
|
|
|
|
nd->last_type, LOOKUP_FOLLOW);
|
2011-03-15 09:20:34 +07:00
|
|
|
put_link(nd, &link, cookie);
|
2011-03-15 08:54:55 +07:00
|
|
|
} while (res > 0);
|
|
|
|
|
|
|
|
current->link_count--;
|
|
|
|
nd->depth--;
|
|
|
|
return res;
|
|
|
|
}
|
|
|
|
|
2011-08-07 12:45:50 +07:00
|
|
|
/*
|
|
|
|
* We really don't want to look at inode->i_op->lookup
|
|
|
|
* when we don't have to. So we keep a cache bit in
|
|
|
|
* the inode ->i_opflags field that says "yes, we can
|
|
|
|
* do lookup on this inode".
|
|
|
|
*/
|
|
|
|
static inline int can_lookup(struct inode *inode)
|
|
|
|
{
|
|
|
|
if (likely(inode->i_opflags & IOP_LOOKUP))
|
|
|
|
return 1;
|
|
|
|
if (likely(!inode->i_op->lookup))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/* We do this once for the lifetime of the inode */
|
|
|
|
spin_lock(&inode->i_lock);
|
|
|
|
inode->i_opflags |= IOP_LOOKUP;
|
|
|
|
spin_unlock(&inode->i_lock);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2012-03-07 02:16:17 +07:00
|
|
|
/*
|
|
|
|
* We can do the critical dentry name comparison and hashing
|
|
|
|
* operations one word at a time, but we are limited to:
|
|
|
|
*
|
|
|
|
* - Architectures with fast unaligned word accesses. We could
|
|
|
|
* do a "get_unaligned()" if this helps and is sufficiently
|
|
|
|
* fast.
|
|
|
|
*
|
|
|
|
* - Little-endian machines (so that we can generate the mask
|
|
|
|
* of low bytes efficiently). Again, we *could* do a byte
|
|
|
|
* swapping load on big-endian architectures if that is not
|
|
|
|
* expensive enough to make the optimization worthless.
|
|
|
|
*
|
|
|
|
* - non-CONFIG_DEBUG_PAGEALLOC configurations (so that we
|
|
|
|
* do not trap on the (extremely unlikely) case of a page
|
|
|
|
* crossing operation.
|
|
|
|
*
|
|
|
|
* - Furthermore, we need an efficient 64-bit compile for the
|
|
|
|
* 64-bit case in order to generate the "number of bytes in
|
|
|
|
* the final mask". Again, that could be replaced with a
|
|
|
|
* efficient population count instruction or similar.
|
|
|
|
*/
|
|
|
|
#ifdef CONFIG_DCACHE_WORD_ACCESS
|
|
|
|
|
2012-04-07 03:54:56 +07:00
|
|
|
#include <asm/word-at-a-time.h>
|
2012-03-07 02:16:17 +07:00
|
|
|
|
2012-04-07 03:54:56 +07:00
|
|
|
#ifdef CONFIG_64BIT
|
2012-03-07 02:16:17 +07:00
|
|
|
|
|
|
|
static inline unsigned int fold_hash(unsigned long hash)
|
|
|
|
{
|
|
|
|
hash += hash >> (8*sizeof(int));
|
|
|
|
return hash;
|
|
|
|
}
|
|
|
|
|
|
|
|
#else /* 32-bit case */
|
|
|
|
|
|
|
|
#define fold_hash(x) (x)
|
|
|
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
unsigned int full_name_hash(const unsigned char *name, unsigned int len)
|
|
|
|
{
|
|
|
|
unsigned long a, mask;
|
|
|
|
unsigned long hash = 0;
|
|
|
|
|
|
|
|
for (;;) {
|
2012-05-04 00:16:43 +07:00
|
|
|
a = load_unaligned_zeropad(name);
|
2012-03-07 02:16:17 +07:00
|
|
|
if (len < sizeof(unsigned long))
|
|
|
|
break;
|
|
|
|
hash += a;
|
2012-03-23 04:59:52 +07:00
|
|
|
hash *= 9;
|
2012-03-07 02:16:17 +07:00
|
|
|
name += sizeof(unsigned long);
|
|
|
|
len -= sizeof(unsigned long);
|
|
|
|
if (!len)
|
|
|
|
goto done;
|
|
|
|
}
|
|
|
|
mask = ~(~0ul << len*8);
|
|
|
|
hash += mask & a;
|
|
|
|
done:
|
|
|
|
return fold_hash(hash);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(full_name_hash);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Calculate the length and hash of the path component, and
|
|
|
|
* return the length of the component;
|
|
|
|
*/
|
|
|
|
static inline unsigned long hash_name(const char *name, unsigned int *hashp)
|
|
|
|
{
|
word-at-a-time: make the interfaces truly generic
This changes the interfaces in <asm/word-at-a-time.h> to be a bit more
complicated, but a lot more generic.
In particular, it allows us to really do the operations efficiently on
both little-endian and big-endian machines, pretty much regardless of
machine details. For example, if you can rely on a fast population
count instruction on your architecture, this will allow you to make your
optimized <asm/word-at-a-time.h> file with that.
NOTE! The "generic" version in include/asm-generic/word-at-a-time.h is
not truly generic, it actually only works on big-endian. Why? Because
on little-endian the generic algorithms are wasteful, since you can
inevitably do better. The x86 implementation is an example of that.
(The only truly non-generic part of the asm-generic implementation is
the "find_zero()" function, and you could make a little-endian version
of it. And if the Kbuild infrastructure allowed us to pick a particular
header file, that would be lovely)
The <asm/word-at-a-time.h> functions are as follows:
- WORD_AT_A_TIME_CONSTANTS: specific constants that the algorithm
uses.
- has_zero(): take a word, and determine if it has a zero byte in it.
It gets the word, the pointer to the constant pool, and a pointer to
an intermediate "data" field it can set.
This is the "quick-and-dirty" zero tester: it's what is run inside
the hot loops.
- "prep_zero_mask()": take the word, the data that has_zero() produced,
and the constant pool, and generate an *exact* mask of which byte had
the first zero. This is run directly *outside* the loop, and allows
the "has_zero()" function to answer the "is there a zero byte"
question without necessarily getting exactly *which* byte is the
first one to contain a zero.
If you do multiple byte lookups concurrently (eg "hash_name()", which
looks for both NUL and '/' bytes), after you've done the prep_zero_mask()
phase, the result of those can be or'ed together to get the "either
or" case.
- The result from "prep_zero_mask()" can then be fed into "find_zero()"
(to find the byte offset of the first byte that was zero) or into
"zero_bytemask()" (to find the bytemask of the bytes preceding the
zero byte).
The existence of zero_bytemask() is optional, and is not necessary
for the normal string routines. But dentry name hashing needs it, so
if you enable DENTRY_WORD_AT_A_TIME you need to expose it.
This changes the generic strncpy_from_user() function and the dentry
hashing functions to use these modified word-at-a-time interfaces. This
gets us back to the optimized state of the x86 strncpy that we lost in
the previous commit when moving over to the generic version.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-05-27 00:43:17 +07:00
|
|
|
unsigned long a, b, adata, bdata, mask, hash, len;
|
|
|
|
const struct word_at_a_time constants = WORD_AT_A_TIME_CONSTANTS;
|
2012-03-07 02:16:17 +07:00
|
|
|
|
|
|
|
hash = a = 0;
|
|
|
|
len = -sizeof(unsigned long);
|
|
|
|
do {
|
|
|
|
hash = (hash + a) * 9;
|
|
|
|
len += sizeof(unsigned long);
|
2012-05-04 00:16:43 +07:00
|
|
|
a = load_unaligned_zeropad(name+len);
|
word-at-a-time: make the interfaces truly generic
This changes the interfaces in <asm/word-at-a-time.h> to be a bit more
complicated, but a lot more generic.
In particular, it allows us to really do the operations efficiently on
both little-endian and big-endian machines, pretty much regardless of
machine details. For example, if you can rely on a fast population
count instruction on your architecture, this will allow you to make your
optimized <asm/word-at-a-time.h> file with that.
NOTE! The "generic" version in include/asm-generic/word-at-a-time.h is
not truly generic, it actually only works on big-endian. Why? Because
on little-endian the generic algorithms are wasteful, since you can
inevitably do better. The x86 implementation is an example of that.
(The only truly non-generic part of the asm-generic implementation is
the "find_zero()" function, and you could make a little-endian version
of it. And if the Kbuild infrastructure allowed us to pick a particular
header file, that would be lovely)
The <asm/word-at-a-time.h> functions are as follows:
- WORD_AT_A_TIME_CONSTANTS: specific constants that the algorithm
uses.
- has_zero(): take a word, and determine if it has a zero byte in it.
It gets the word, the pointer to the constant pool, and a pointer to
an intermediate "data" field it can set.
This is the "quick-and-dirty" zero tester: it's what is run inside
the hot loops.
- "prep_zero_mask()": take the word, the data that has_zero() produced,
and the constant pool, and generate an *exact* mask of which byte had
the first zero. This is run directly *outside* the loop, and allows
the "has_zero()" function to answer the "is there a zero byte"
question without necessarily getting exactly *which* byte is the
first one to contain a zero.
If you do multiple byte lookups concurrently (eg "hash_name()", which
looks for both NUL and '/' bytes), after you've done the prep_zero_mask()
phase, the result of those can be or'ed together to get the "either
or" case.
- The result from "prep_zero_mask()" can then be fed into "find_zero()"
(to find the byte offset of the first byte that was zero) or into
"zero_bytemask()" (to find the bytemask of the bytes preceding the
zero byte).
The existence of zero_bytemask() is optional, and is not necessary
for the normal string routines. But dentry name hashing needs it, so
if you enable DENTRY_WORD_AT_A_TIME you need to expose it.
This changes the generic strncpy_from_user() function and the dentry
hashing functions to use these modified word-at-a-time interfaces. This
gets us back to the optimized state of the x86 strncpy that we lost in
the previous commit when moving over to the generic version.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-05-27 00:43:17 +07:00
|
|
|
b = a ^ REPEAT_BYTE('/');
|
|
|
|
} while (!(has_zero(a, &adata, &constants) | has_zero(b, &bdata, &constants)));
|
|
|
|
|
|
|
|
adata = prep_zero_mask(a, adata, &constants);
|
|
|
|
bdata = prep_zero_mask(b, bdata, &constants);
|
|
|
|
|
|
|
|
mask = create_zero_mask(adata | bdata);
|
|
|
|
|
|
|
|
hash += a & zero_bytemask(mask);
|
2012-03-07 02:16:17 +07:00
|
|
|
*hashp = fold_hash(hash);
|
|
|
|
|
word-at-a-time: make the interfaces truly generic
This changes the interfaces in <asm/word-at-a-time.h> to be a bit more
complicated, but a lot more generic.
In particular, it allows us to really do the operations efficiently on
both little-endian and big-endian machines, pretty much regardless of
machine details. For example, if you can rely on a fast population
count instruction on your architecture, this will allow you to make your
optimized <asm/word-at-a-time.h> file with that.
NOTE! The "generic" version in include/asm-generic/word-at-a-time.h is
not truly generic, it actually only works on big-endian. Why? Because
on little-endian the generic algorithms are wasteful, since you can
inevitably do better. The x86 implementation is an example of that.
(The only truly non-generic part of the asm-generic implementation is
the "find_zero()" function, and you could make a little-endian version
of it. And if the Kbuild infrastructure allowed us to pick a particular
header file, that would be lovely)
The <asm/word-at-a-time.h> functions are as follows:
- WORD_AT_A_TIME_CONSTANTS: specific constants that the algorithm
uses.
- has_zero(): take a word, and determine if it has a zero byte in it.
It gets the word, the pointer to the constant pool, and a pointer to
an intermediate "data" field it can set.
This is the "quick-and-dirty" zero tester: it's what is run inside
the hot loops.
- "prep_zero_mask()": take the word, the data that has_zero() produced,
and the constant pool, and generate an *exact* mask of which byte had
the first zero. This is run directly *outside* the loop, and allows
the "has_zero()" function to answer the "is there a zero byte"
question without necessarily getting exactly *which* byte is the
first one to contain a zero.
If you do multiple byte lookups concurrently (eg "hash_name()", which
looks for both NUL and '/' bytes), after you've done the prep_zero_mask()
phase, the result of those can be or'ed together to get the "either
or" case.
- The result from "prep_zero_mask()" can then be fed into "find_zero()"
(to find the byte offset of the first byte that was zero) or into
"zero_bytemask()" (to find the bytemask of the bytes preceding the
zero byte).
The existence of zero_bytemask() is optional, and is not necessary
for the normal string routines. But dentry name hashing needs it, so
if you enable DENTRY_WORD_AT_A_TIME you need to expose it.
This changes the generic strncpy_from_user() function and the dentry
hashing functions to use these modified word-at-a-time interfaces. This
gets us back to the optimized state of the x86 strncpy that we lost in
the previous commit when moving over to the generic version.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-05-27 00:43:17 +07:00
|
|
|
return len + find_zero(mask);
|
2012-03-07 02:16:17 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
#else
|
|
|
|
|
2012-03-03 05:32:59 +07:00
|
|
|
unsigned int full_name_hash(const unsigned char *name, unsigned int len)
|
|
|
|
{
|
|
|
|
unsigned long hash = init_name_hash();
|
|
|
|
while (len--)
|
|
|
|
hash = partial_name_hash(*name++, hash);
|
|
|
|
return end_name_hash(hash);
|
|
|
|
}
|
2012-03-03 10:40:57 +07:00
|
|
|
EXPORT_SYMBOL(full_name_hash);
|
2012-03-03 05:32:59 +07:00
|
|
|
|
2012-03-03 05:49:24 +07:00
|
|
|
/*
|
|
|
|
* We know there's a real path component here of at least
|
|
|
|
* one character.
|
|
|
|
*/
|
|
|
|
static inline unsigned long hash_name(const char *name, unsigned int *hashp)
|
|
|
|
{
|
|
|
|
unsigned long hash = init_name_hash();
|
|
|
|
unsigned long len = 0, c;
|
|
|
|
|
|
|
|
c = (unsigned char)*name;
|
|
|
|
do {
|
|
|
|
len++;
|
|
|
|
hash = partial_name_hash(c, hash);
|
|
|
|
c = (unsigned char)name[len];
|
|
|
|
} while (c && c != '/');
|
|
|
|
*hashp = end_name_hash(hash);
|
|
|
|
return len;
|
|
|
|
}
|
|
|
|
|
2012-03-07 02:16:17 +07:00
|
|
|
#endif
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* Name resolution.
|
2005-04-29 22:00:17 +07:00
|
|
|
* This is the basic name resolution function, turning a pathname into
|
|
|
|
* the final dentry. We expect 'base' to be positive and a directory.
|
2005-04-17 05:20:36 +07:00
|
|
|
*
|
2005-04-29 22:00:17 +07:00
|
|
|
* Returns 0 and nd will have valid dentry and mnt on success.
|
|
|
|
* Returns error and drops reference to input namei data on failure.
|
2005-04-17 05:20:36 +07:00
|
|
|
*/
|
2009-08-09 04:41:57 +07:00
|
|
|
static int link_path_walk(const char *name, struct nameidata *nd)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
struct path next;
|
|
|
|
int err;
|
|
|
|
|
|
|
|
while (*name=='/')
|
|
|
|
name++;
|
|
|
|
if (!*name)
|
2011-02-23 08:56:27 +07:00
|
|
|
return 0;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
/* At this point we know we have a real path component. */
|
|
|
|
for(;;) {
|
|
|
|
struct qstr this;
|
2012-03-03 05:49:24 +07:00
|
|
|
long len;
|
2011-02-23 03:10:03 +07:00
|
|
|
int type;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2011-02-22 09:34:47 +07:00
|
|
|
err = may_lookup(nd);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (err)
|
|
|
|
break;
|
|
|
|
|
2012-03-03 05:49:24 +07:00
|
|
|
len = hash_name(name, &this.hash);
|
2005-04-17 05:20:36 +07:00
|
|
|
this.name = name;
|
2012-03-03 05:49:24 +07:00
|
|
|
this.len = len;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2011-02-23 03:10:03 +07:00
|
|
|
type = LAST_NORM;
|
2012-03-03 05:49:24 +07:00
|
|
|
if (name[0] == '.') switch (len) {
|
2011-02-23 03:10:03 +07:00
|
|
|
case 2:
|
2012-03-03 05:49:24 +07:00
|
|
|
if (name[1] == '.') {
|
2011-02-23 03:10:03 +07:00
|
|
|
type = LAST_DOTDOT;
|
2011-02-23 03:50:10 +07:00
|
|
|
nd->flags |= LOOKUP_JUMPED;
|
|
|
|
}
|
2011-02-23 03:10:03 +07:00
|
|
|
break;
|
|
|
|
case 1:
|
|
|
|
type = LAST_DOT;
|
|
|
|
}
|
2011-03-09 02:17:44 +07:00
|
|
|
if (likely(type == LAST_NORM)) {
|
|
|
|
struct dentry *parent = nd->path.dentry;
|
2011-02-23 03:50:10 +07:00
|
|
|
nd->flags &= ~LOOKUP_JUMPED;
|
2011-03-09 02:17:44 +07:00
|
|
|
if (unlikely(parent->d_flags & DCACHE_OP_HASH)) {
|
|
|
|
err = parent->d_op->d_hash(parent, nd->inode,
|
|
|
|
&this);
|
|
|
|
if (err < 0)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
2011-02-23 03:10:03 +07:00
|
|
|
|
2012-03-03 05:49:24 +07:00
|
|
|
if (!name[len])
|
2005-04-17 05:20:36 +07:00
|
|
|
goto last_component;
|
2012-03-03 05:49:24 +07:00
|
|
|
/*
|
|
|
|
* If it wasn't NUL, we know it was '/'. Skip that
|
|
|
|
* slash, and continue until no more slashes.
|
|
|
|
*/
|
|
|
|
do {
|
|
|
|
len++;
|
|
|
|
} while (unlikely(name[len] == '/'));
|
|
|
|
if (!name[len])
|
2011-03-15 08:54:55 +07:00
|
|
|
goto last_component;
|
2012-03-03 05:49:24 +07:00
|
|
|
name += len;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2011-03-14 06:58:58 +07:00
|
|
|
err = walk_component(nd, &next, &this, type, LOOKUP_FOLLOW);
|
|
|
|
if (err < 0)
|
|
|
|
return err;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2011-03-14 06:58:58 +07:00
|
|
|
if (err) {
|
2011-03-15 08:54:55 +07:00
|
|
|
err = nested_symlink(&next, nd);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (err)
|
2011-03-05 02:39:30 +07:00
|
|
|
return err;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
}
|
2011-08-07 12:45:50 +07:00
|
|
|
if (can_lookup(nd->inode))
|
|
|
|
continue;
|
2005-04-17 05:20:36 +07:00
|
|
|
err = -ENOTDIR;
|
2011-08-07 12:45:50 +07:00
|
|
|
break;
|
2005-04-17 05:20:36 +07:00
|
|
|
/* here ends the main loop */
|
|
|
|
|
|
|
|
last_component:
|
2011-03-15 08:54:55 +07:00
|
|
|
nd->last = this;
|
|
|
|
nd->last_type = type;
|
2011-02-23 08:56:27 +07:00
|
|
|
return 0;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
2011-03-05 02:44:37 +07:00
|
|
|
terminate_walk(nd);
|
2005-04-17 05:20:36 +07:00
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2011-03-06 09:12:22 +07:00
|
|
|
static int path_init(int dfd, const char *name, unsigned int flags,
|
|
|
|
struct nameidata *nd, struct file **fp)
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
{
|
|
|
|
int retval = 0;
|
|
|
|
int fput_needed;
|
|
|
|
struct file *file;
|
|
|
|
|
|
|
|
nd->last_type = LAST_ROOT; /* if there are only slashes... */
|
2011-02-23 03:50:10 +07:00
|
|
|
nd->flags = flags | LOOKUP_JUMPED;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
nd->depth = 0;
|
2011-03-10 11:04:47 +07:00
|
|
|
if (flags & LOOKUP_ROOT) {
|
|
|
|
struct inode *inode = nd->root.dentry->d_inode;
|
2011-03-12 00:08:24 +07:00
|
|
|
if (*name) {
|
|
|
|
if (!inode->i_op->lookup)
|
|
|
|
return -ENOTDIR;
|
|
|
|
retval = inode_permission(inode, MAY_EXEC);
|
|
|
|
if (retval)
|
|
|
|
return retval;
|
|
|
|
}
|
2011-03-10 11:04:47 +07:00
|
|
|
nd->path = nd->root;
|
|
|
|
nd->inode = inode;
|
|
|
|
if (flags & LOOKUP_RCU) {
|
2012-05-08 11:02:02 +07:00
|
|
|
br_read_lock(&vfsmount_lock);
|
2011-03-10 11:04:47 +07:00
|
|
|
rcu_read_lock();
|
|
|
|
nd->seq = __read_seqcount_begin(&nd->path.dentry->d_seq);
|
|
|
|
} else {
|
|
|
|
path_get(&nd->path);
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
nd->root.mnt = NULL;
|
|
|
|
|
|
|
|
if (*name=='/') {
|
2011-02-23 02:02:58 +07:00
|
|
|
if (flags & LOOKUP_RCU) {
|
2012-05-08 11:02:02 +07:00
|
|
|
br_read_lock(&vfsmount_lock);
|
2011-02-23 02:02:58 +07:00
|
|
|
rcu_read_lock();
|
|
|
|
set_root_rcu(nd);
|
|
|
|
} else {
|
|
|
|
set_root(nd);
|
|
|
|
path_get(&nd->root);
|
|
|
|
}
|
|
|
|
nd->path = nd->root;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
} else if (dfd == AT_FDCWD) {
|
2011-02-23 02:02:58 +07:00
|
|
|
if (flags & LOOKUP_RCU) {
|
|
|
|
struct fs_struct *fs = current->fs;
|
|
|
|
unsigned seq;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
|
2012-05-08 11:02:02 +07:00
|
|
|
br_read_lock(&vfsmount_lock);
|
2011-02-23 02:02:58 +07:00
|
|
|
rcu_read_lock();
|
2011-01-07 13:49:53 +07:00
|
|
|
|
2011-02-23 02:02:58 +07:00
|
|
|
do {
|
|
|
|
seq = read_seqcount_begin(&fs->seq);
|
|
|
|
nd->path = fs->pwd;
|
|
|
|
nd->seq = __read_seqcount_begin(&nd->path.dentry->d_seq);
|
|
|
|
} while (read_seqcount_retry(&fs->seq, seq));
|
|
|
|
} else {
|
|
|
|
get_fs_pwd(current->fs, &nd->path);
|
|
|
|
}
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
} else {
|
|
|
|
struct dentry *dentry;
|
|
|
|
|
New kind of open files - "location only".
New flag for open(2) - O_PATH. Semantics:
* pathname is resolved, but the file itself is _NOT_ opened
as far as filesystem is concerned.
* almost all operations on the resulting descriptors shall
fail with -EBADF. Exceptions are:
1) operations on descriptors themselves (i.e.
close(), dup(), dup2(), dup3(), fcntl(fd, F_DUPFD),
fcntl(fd, F_DUPFD_CLOEXEC, ...), fcntl(fd, F_GETFD),
fcntl(fd, F_SETFD, ...))
2) fcntl(fd, F_GETFL), for a common non-destructive way to
check if descriptor is open
3) "dfd" arguments of ...at(2) syscalls, i.e. the starting
points of pathname resolution
* closing such descriptor does *NOT* affect dnotify or
posix locks.
* permissions are checked as usual along the way to file;
no permission checks are applied to the file itself. Of course,
giving such thing to syscall will result in permission checks (at
the moment it means checking that starting point of ....at() is
a directory and caller has exec permissions on it).
fget() and fget_light() return NULL on such descriptors; use of
fget_raw() and fget_raw_light() is needed to get them. That protects
existing code from dealing with those things.
There are two things still missing (they come in the next commits):
one is handling of symlinks (right now we refuse to open them that
way; see the next commit for semantics related to those) and another
is descriptor passing via SCM_RIGHTS datagrams.
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-03-13 14:51:11 +07:00
|
|
|
file = fget_raw_light(dfd, &fput_needed);
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
retval = -EBADF;
|
|
|
|
if (!file)
|
|
|
|
goto out_fail;
|
|
|
|
|
|
|
|
dentry = file->f_path.dentry;
|
|
|
|
|
2011-03-15 05:56:51 +07:00
|
|
|
if (*name) {
|
|
|
|
retval = -ENOTDIR;
|
|
|
|
if (!S_ISDIR(dentry->d_inode->i_mode))
|
|
|
|
goto fput_fail;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
|
2011-06-21 06:57:03 +07:00
|
|
|
retval = inode_permission(dentry->d_inode, MAY_EXEC);
|
2011-03-15 05:56:51 +07:00
|
|
|
if (retval)
|
|
|
|
goto fput_fail;
|
|
|
|
}
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
|
|
|
|
nd->path = file->f_path;
|
2011-02-23 02:02:58 +07:00
|
|
|
if (flags & LOOKUP_RCU) {
|
|
|
|
if (fput_needed)
|
2011-03-06 09:12:22 +07:00
|
|
|
*fp = file;
|
2011-02-23 02:02:58 +07:00
|
|
|
nd->seq = __read_seqcount_begin(&nd->path.dentry->d_seq);
|
2012-05-08 11:02:02 +07:00
|
|
|
br_read_lock(&vfsmount_lock);
|
2011-02-23 02:02:58 +07:00
|
|
|
rcu_read_lock();
|
|
|
|
} else {
|
|
|
|
path_get(&file->f_path);
|
|
|
|
fput_light(file, fput_needed);
|
|
|
|
}
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
nd->inode = nd->path.dentry->d_inode;
|
2009-04-07 22:44:16 +07:00
|
|
|
return 0;
|
2007-05-09 16:33:41 +07:00
|
|
|
|
2009-04-07 22:44:16 +07:00
|
|
|
fput_fail:
|
|
|
|
fput_light(file, fput_needed);
|
|
|
|
out_fail:
|
|
|
|
return retval;
|
|
|
|
}
|
|
|
|
|
2011-03-15 06:54:59 +07:00
|
|
|
static inline int lookup_last(struct nameidata *nd, struct path *path)
|
|
|
|
{
|
|
|
|
if (nd->last_type == LAST_NORM && nd->last.name[nd->last.len])
|
|
|
|
nd->flags |= LOOKUP_FOLLOW | LOOKUP_DIRECTORY;
|
|
|
|
|
|
|
|
nd->flags &= ~LOOKUP_PARENT;
|
|
|
|
return walk_component(nd, path, &nd->last, nd->last_type,
|
|
|
|
nd->flags & LOOKUP_FOLLOW);
|
|
|
|
}
|
|
|
|
|
2009-04-07 22:44:16 +07:00
|
|
|
/* Returns 0 and nd will be valid on success; Retuns error, otherwise. */
|
2011-02-22 11:38:09 +07:00
|
|
|
static int path_lookupat(int dfd, const char *name,
|
2009-04-07 22:44:16 +07:00
|
|
|
unsigned int flags, struct nameidata *nd)
|
|
|
|
{
|
2011-03-06 09:12:22 +07:00
|
|
|
struct file *base = NULL;
|
2011-03-15 06:54:59 +07:00
|
|
|
struct path path;
|
|
|
|
int err;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Path walking is largely split up into 2 different synchronisation
|
|
|
|
* schemes, rcu-walk and ref-walk (explained in
|
|
|
|
* Documentation/filesystems/path-lookup.txt). These share much of the
|
|
|
|
* path walk code, but some things particularly setup, cleanup, and
|
|
|
|
* following mounts are sufficiently divergent that functions are
|
|
|
|
* duplicated. Typically there is a function foo(), and its RCU
|
|
|
|
* analogue, foo_rcu().
|
|
|
|
*
|
|
|
|
* -ECHILD is the error number of choice (just to avoid clashes) that
|
|
|
|
* is returned if some aspect of an rcu-walk fails. Such an error must
|
|
|
|
* be handled by restarting a traditional ref-walk (which will always
|
|
|
|
* be able to complete).
|
|
|
|
*/
|
2011-03-15 06:54:59 +07:00
|
|
|
err = path_init(dfd, name, flags | LOOKUP_PARENT, nd, &base);
|
2011-02-22 11:38:09 +07:00
|
|
|
|
2011-03-15 06:54:59 +07:00
|
|
|
if (unlikely(err))
|
|
|
|
return err;
|
2011-02-22 11:38:09 +07:00
|
|
|
|
|
|
|
current->total_link_count = 0;
|
2011-03-15 06:54:59 +07:00
|
|
|
err = link_path_walk(name, nd);
|
|
|
|
|
|
|
|
if (!err && !(flags & LOOKUP_PARENT)) {
|
|
|
|
err = lookup_last(nd, &path);
|
|
|
|
while (err > 0) {
|
|
|
|
void *cookie;
|
|
|
|
struct path link = path;
|
|
|
|
nd->flags |= LOOKUP_PARENT;
|
2011-03-15 09:20:34 +07:00
|
|
|
err = follow_link(&link, nd, &cookie);
|
2011-03-15 06:54:59 +07:00
|
|
|
if (!err)
|
|
|
|
err = lookup_last(nd, &path);
|
2011-03-15 09:20:34 +07:00
|
|
|
put_link(nd, &link, cookie);
|
2011-03-15 06:54:59 +07:00
|
|
|
}
|
|
|
|
}
|
2011-02-22 11:38:09 +07:00
|
|
|
|
2011-03-25 22:00:12 +07:00
|
|
|
if (!err)
|
|
|
|
err = complete_walk(nd);
|
2011-03-15 06:54:59 +07:00
|
|
|
|
|
|
|
if (!err && nd->flags & LOOKUP_DIRECTORY) {
|
|
|
|
if (!nd->inode->i_op->lookup) {
|
|
|
|
path_put(&nd->path);
|
2011-03-23 20:56:30 +07:00
|
|
|
err = -ENOTDIR;
|
2011-03-15 06:54:59 +07:00
|
|
|
}
|
|
|
|
}
|
2011-02-23 03:50:10 +07:00
|
|
|
|
2011-03-06 09:12:22 +07:00
|
|
|
if (base)
|
|
|
|
fput(base);
|
2011-02-22 11:38:09 +07:00
|
|
|
|
2011-03-10 11:04:47 +07:00
|
|
|
if (nd->root.mnt && !(nd->flags & LOOKUP_ROOT)) {
|
2009-04-07 22:49:53 +07:00
|
|
|
path_put(&nd->root);
|
|
|
|
nd->root.mnt = NULL;
|
|
|
|
}
|
2011-03-15 06:54:59 +07:00
|
|
|
return err;
|
2011-02-22 11:38:09 +07:00
|
|
|
}
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
|
2011-02-22 11:38:09 +07:00
|
|
|
static int do_path_lookup(int dfd, const char *name,
|
|
|
|
unsigned int flags, struct nameidata *nd)
|
|
|
|
{
|
|
|
|
int retval = path_lookupat(dfd, name, flags | LOOKUP_RCU, nd);
|
|
|
|
if (unlikely(retval == -ECHILD))
|
|
|
|
retval = path_lookupat(dfd, name, flags, nd);
|
|
|
|
if (unlikely(retval == -ESTALE))
|
|
|
|
retval = path_lookupat(dfd, name, flags | LOOKUP_REVAL, nd);
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
|
|
|
|
if (likely(!retval)) {
|
|
|
|
if (unlikely(!audit_dummy_context())) {
|
|
|
|
if (nd->path.dentry && nd->inode)
|
|
|
|
audit_inode(name, nd->path.dentry);
|
|
|
|
}
|
|
|
|
}
|
2006-02-05 14:28:02 +07:00
|
|
|
return retval;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2011-02-17 03:15:47 +07:00
|
|
|
int kern_path_parent(const char *name, struct nameidata *nd)
|
2006-01-19 08:43:53 +07:00
|
|
|
{
|
2011-02-17 03:15:47 +07:00
|
|
|
return do_path_lookup(AT_FDCWD, name, LOOKUP_PARENT, nd);
|
2006-01-19 08:43:53 +07:00
|
|
|
}
|
|
|
|
|
2008-08-02 11:49:18 +07:00
|
|
|
int kern_path(const char *name, unsigned int flags, struct path *path)
|
|
|
|
{
|
|
|
|
struct nameidata nd;
|
|
|
|
int res = do_path_lookup(AT_FDCWD, name, flags, &nd);
|
|
|
|
if (!res)
|
|
|
|
*path = nd.path;
|
|
|
|
return res;
|
|
|
|
}
|
|
|
|
|
fs: introduce vfs_path_lookup
Stackable file systems, among others, frequently need to lookup paths or
path components starting from an arbitrary point in the namespace
(identified by a dentry and a vfsmount). Currently, such file systems use
lookup_one_len, which is frowned upon [1] as it does not pass the lookup
intent along; not passing a lookup intent, for example, can trigger BUG_ON's
when stacking on top of NFSv4.
The first patch introduces a new lookup function to allow lookup starting
from an arbitrary point in the namespace. This approach has been suggested
by Christoph Hellwig [2].
The second patch changes sunrpc to use vfs_path_lookup.
The third patch changes nfsctl.c to use vfs_path_lookup.
The fourth patch marks link_path_walk static.
The fifth, and last patch, unexports path_walk because it is no longer
unnecessary to call it directly, and using the new vfs_path_lookup is
cleaner.
For example, the following snippet of code, looks up "some/path/component"
in a directory pointed to by parent_{dentry,vfsmnt}:
err = vfs_path_lookup(parent_dentry, parent_vfsmnt,
"some/path/component", 0, &nd);
if (!err) {
/* exits */
...
/* once done, release the references */
path_release(&nd);
} else if (err == -ENOENT) {
/* doesn't exist */
} else {
/* other error */
}
VFS functions such as lookup_create can be used on the nameidata structure
to pass the create intent to the file system.
Signed-off-by: Josef 'Jeff' Sipek <jsipek@cs.sunysb.edu>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Acked-by: Christoph Hellwig <hch@lst.de>
Cc: Trond Myklebust <trond.myklebust@fys.uio.no>
Cc: Neil Brown <neilb@suse.de>
Cc: Michael Halcrow <mhalcrow@us.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-19 15:48:18 +07:00
|
|
|
/**
|
|
|
|
* vfs_path_lookup - lookup a file path relative to a dentry-vfsmount pair
|
|
|
|
* @dentry: pointer to dentry of the base directory
|
|
|
|
* @mnt: pointer to vfs mount of the base directory
|
|
|
|
* @name: pointer to file name
|
|
|
|
* @flags: lookup flags
|
2011-06-28 04:00:37 +07:00
|
|
|
* @path: pointer to struct path to fill
|
fs: introduce vfs_path_lookup
Stackable file systems, among others, frequently need to lookup paths or
path components starting from an arbitrary point in the namespace
(identified by a dentry and a vfsmount). Currently, such file systems use
lookup_one_len, which is frowned upon [1] as it does not pass the lookup
intent along; not passing a lookup intent, for example, can trigger BUG_ON's
when stacking on top of NFSv4.
The first patch introduces a new lookup function to allow lookup starting
from an arbitrary point in the namespace. This approach has been suggested
by Christoph Hellwig [2].
The second patch changes sunrpc to use vfs_path_lookup.
The third patch changes nfsctl.c to use vfs_path_lookup.
The fourth patch marks link_path_walk static.
The fifth, and last patch, unexports path_walk because it is no longer
unnecessary to call it directly, and using the new vfs_path_lookup is
cleaner.
For example, the following snippet of code, looks up "some/path/component"
in a directory pointed to by parent_{dentry,vfsmnt}:
err = vfs_path_lookup(parent_dentry, parent_vfsmnt,
"some/path/component", 0, &nd);
if (!err) {
/* exits */
...
/* once done, release the references */
path_release(&nd);
} else if (err == -ENOENT) {
/* doesn't exist */
} else {
/* other error */
}
VFS functions such as lookup_create can be used on the nameidata structure
to pass the create intent to the file system.
Signed-off-by: Josef 'Jeff' Sipek <jsipek@cs.sunysb.edu>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Acked-by: Christoph Hellwig <hch@lst.de>
Cc: Trond Myklebust <trond.myklebust@fys.uio.no>
Cc: Neil Brown <neilb@suse.de>
Cc: Michael Halcrow <mhalcrow@us.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-19 15:48:18 +07:00
|
|
|
*/
|
|
|
|
int vfs_path_lookup(struct dentry *dentry, struct vfsmount *mnt,
|
|
|
|
const char *name, unsigned int flags,
|
2011-06-28 04:00:37 +07:00
|
|
|
struct path *path)
|
fs: introduce vfs_path_lookup
Stackable file systems, among others, frequently need to lookup paths or
path components starting from an arbitrary point in the namespace
(identified by a dentry and a vfsmount). Currently, such file systems use
lookup_one_len, which is frowned upon [1] as it does not pass the lookup
intent along; not passing a lookup intent, for example, can trigger BUG_ON's
when stacking on top of NFSv4.
The first patch introduces a new lookup function to allow lookup starting
from an arbitrary point in the namespace. This approach has been suggested
by Christoph Hellwig [2].
The second patch changes sunrpc to use vfs_path_lookup.
The third patch changes nfsctl.c to use vfs_path_lookup.
The fourth patch marks link_path_walk static.
The fifth, and last patch, unexports path_walk because it is no longer
unnecessary to call it directly, and using the new vfs_path_lookup is
cleaner.
For example, the following snippet of code, looks up "some/path/component"
in a directory pointed to by parent_{dentry,vfsmnt}:
err = vfs_path_lookup(parent_dentry, parent_vfsmnt,
"some/path/component", 0, &nd);
if (!err) {
/* exits */
...
/* once done, release the references */
path_release(&nd);
} else if (err == -ENOENT) {
/* doesn't exist */
} else {
/* other error */
}
VFS functions such as lookup_create can be used on the nameidata structure
to pass the create intent to the file system.
Signed-off-by: Josef 'Jeff' Sipek <jsipek@cs.sunysb.edu>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Acked-by: Christoph Hellwig <hch@lst.de>
Cc: Trond Myklebust <trond.myklebust@fys.uio.no>
Cc: Neil Brown <neilb@suse.de>
Cc: Michael Halcrow <mhalcrow@us.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-19 15:48:18 +07:00
|
|
|
{
|
2011-06-28 04:00:37 +07:00
|
|
|
struct nameidata nd;
|
|
|
|
int err;
|
|
|
|
nd.root.dentry = dentry;
|
|
|
|
nd.root.mnt = mnt;
|
|
|
|
BUG_ON(flags & LOOKUP_PARENT);
|
2011-03-10 11:04:47 +07:00
|
|
|
/* the first argument of do_path_lookup() is ignored with LOOKUP_ROOT */
|
2011-06-28 04:00:37 +07:00
|
|
|
err = do_path_lookup(AT_FDCWD, name, flags | LOOKUP_ROOT, &nd);
|
|
|
|
if (!err)
|
|
|
|
*path = nd.path;
|
|
|
|
return err;
|
fs: introduce vfs_path_lookup
Stackable file systems, among others, frequently need to lookup paths or
path components starting from an arbitrary point in the namespace
(identified by a dentry and a vfsmount). Currently, such file systems use
lookup_one_len, which is frowned upon [1] as it does not pass the lookup
intent along; not passing a lookup intent, for example, can trigger BUG_ON's
when stacking on top of NFSv4.
The first patch introduces a new lookup function to allow lookup starting
from an arbitrary point in the namespace. This approach has been suggested
by Christoph Hellwig [2].
The second patch changes sunrpc to use vfs_path_lookup.
The third patch changes nfsctl.c to use vfs_path_lookup.
The fourth patch marks link_path_walk static.
The fifth, and last patch, unexports path_walk because it is no longer
unnecessary to call it directly, and using the new vfs_path_lookup is
cleaner.
For example, the following snippet of code, looks up "some/path/component"
in a directory pointed to by parent_{dentry,vfsmnt}:
err = vfs_path_lookup(parent_dentry, parent_vfsmnt,
"some/path/component", 0, &nd);
if (!err) {
/* exits */
...
/* once done, release the references */
path_release(&nd);
} else if (err == -ENOENT) {
/* doesn't exist */
} else {
/* other error */
}
VFS functions such as lookup_create can be used on the nameidata structure
to pass the create intent to the file system.
Signed-off-by: Josef 'Jeff' Sipek <jsipek@cs.sunysb.edu>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Acked-by: Christoph Hellwig <hch@lst.de>
Cc: Trond Myklebust <trond.myklebust@fys.uio.no>
Cc: Neil Brown <neilb@suse.de>
Cc: Michael Halcrow <mhalcrow@us.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-19 15:48:18 +07:00
|
|
|
}
|
|
|
|
|
2007-04-26 14:12:05 +07:00
|
|
|
/*
|
|
|
|
* Restricted form of lookup. Doesn't follow links, single-component only,
|
|
|
|
* needs parent already locked. Doesn't follow mounts.
|
|
|
|
* SMP-safe.
|
|
|
|
*/
|
2007-10-17 13:25:38 +07:00
|
|
|
static struct dentry *lookup_hash(struct nameidata *nd)
|
2007-04-26 14:12:05 +07:00
|
|
|
{
|
2008-02-15 10:34:32 +07:00
|
|
|
return __lookup_hash(&nd->last, nd->path.dentry, nd);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2007-10-17 13:25:38 +07:00
|
|
|
/**
|
2008-03-20 07:01:00 +07:00
|
|
|
* lookup_one_len - filesystem helper to lookup single pathname component
|
2007-10-17 13:25:38 +07:00
|
|
|
* @name: pathname component to lookup
|
|
|
|
* @base: base directory to lookup from
|
|
|
|
* @len: maximum length @len should be interpreted to
|
|
|
|
*
|
2008-03-20 07:01:00 +07:00
|
|
|
* Note that this routine is purely a helper for filesystem usage and should
|
|
|
|
* not be called by generic code. Also note that by using this function the
|
2007-10-17 13:25:38 +07:00
|
|
|
* nameidata argument is passed to the filesystem methods and a filesystem
|
|
|
|
* using this helper needs to be prepared for that.
|
|
|
|
*/
|
2007-04-26 14:12:05 +07:00
|
|
|
struct dentry *lookup_one_len(const char *name, struct dentry *base, int len)
|
|
|
|
{
|
|
|
|
struct qstr this;
|
2011-03-08 11:49:20 +07:00
|
|
|
unsigned int c;
|
2012-03-26 17:54:21 +07:00
|
|
|
int err;
|
2007-04-26 14:12:05 +07:00
|
|
|
|
2009-04-21 05:18:37 +07:00
|
|
|
WARN_ON_ONCE(!mutex_is_locked(&base->d_inode->i_mutex));
|
|
|
|
|
2011-03-08 11:49:20 +07:00
|
|
|
this.name = name;
|
|
|
|
this.len = len;
|
2012-03-03 05:32:59 +07:00
|
|
|
this.hash = full_name_hash(name, len);
|
2011-03-08 11:49:20 +07:00
|
|
|
if (!len)
|
|
|
|
return ERR_PTR(-EACCES);
|
|
|
|
|
|
|
|
while (len--) {
|
|
|
|
c = *(const unsigned char *)name++;
|
|
|
|
if (c == '/' || c == '\0')
|
|
|
|
return ERR_PTR(-EACCES);
|
|
|
|
}
|
2011-03-09 02:17:44 +07:00
|
|
|
/*
|
|
|
|
* See if the low-level filesystem might want
|
|
|
|
* to use its own hash..
|
|
|
|
*/
|
|
|
|
if (base->d_flags & DCACHE_OP_HASH) {
|
|
|
|
int err = base->d_op->d_hash(base, base->d_inode, &this);
|
|
|
|
if (err < 0)
|
|
|
|
return ERR_PTR(err);
|
|
|
|
}
|
2007-10-17 13:25:38 +07:00
|
|
|
|
2012-03-26 17:54:21 +07:00
|
|
|
err = inode_permission(base->d_inode, MAY_EXEC);
|
|
|
|
if (err)
|
|
|
|
return ERR_PTR(err);
|
|
|
|
|
2005-11-09 12:35:06 +07:00
|
|
|
return __lookup_hash(&this, base, NULL);
|
2007-04-26 14:12:05 +07:00
|
|
|
}
|
|
|
|
|
2011-11-02 15:44:39 +07:00
|
|
|
int user_path_at_empty(int dfd, const char __user *name, unsigned flags,
|
|
|
|
struct path *path, int *empty)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2008-07-22 20:59:21 +07:00
|
|
|
struct nameidata nd;
|
2011-11-02 15:44:39 +07:00
|
|
|
char *tmp = getname_flags(name, flags, empty);
|
2005-04-17 05:20:36 +07:00
|
|
|
int err = PTR_ERR(tmp);
|
|
|
|
if (!IS_ERR(tmp)) {
|
2008-07-22 20:59:21 +07:00
|
|
|
|
|
|
|
BUG_ON(flags & LOOKUP_PARENT);
|
|
|
|
|
|
|
|
err = do_path_lookup(dfd, tmp, flags, &nd);
|
2005-04-17 05:20:36 +07:00
|
|
|
putname(tmp);
|
2008-07-22 20:59:21 +07:00
|
|
|
if (!err)
|
|
|
|
*path = nd.path;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2011-11-02 15:44:39 +07:00
|
|
|
int user_path_at(int dfd, const char __user *name, unsigned flags,
|
|
|
|
struct path *path)
|
|
|
|
{
|
2012-03-23 06:10:40 +07:00
|
|
|
return user_path_at_empty(dfd, name, flags, path, NULL);
|
2011-11-02 15:44:39 +07:00
|
|
|
}
|
|
|
|
|
2008-07-21 20:32:51 +07:00
|
|
|
static int user_path_parent(int dfd, const char __user *path,
|
|
|
|
struct nameidata *nd, char **name)
|
|
|
|
{
|
|
|
|
char *s = getname(path);
|
|
|
|
int error;
|
|
|
|
|
|
|
|
if (IS_ERR(s))
|
|
|
|
return PTR_ERR(s);
|
|
|
|
|
|
|
|
error = do_path_lookup(dfd, s, LOOKUP_PARENT, nd);
|
|
|
|
if (error)
|
|
|
|
putname(s);
|
|
|
|
else
|
|
|
|
*name = s;
|
|
|
|
|
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* It's inline, so penalty for filesystems that don't use sticky bit is
|
|
|
|
* minimal.
|
|
|
|
*/
|
|
|
|
static inline int check_sticky(struct inode *dir, struct inode *inode)
|
|
|
|
{
|
2012-03-04 12:17:15 +07:00
|
|
|
kuid_t fsuid = current_fsuid();
|
2008-11-14 06:39:05 +07:00
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
if (!(dir->i_mode & S_ISVTX))
|
|
|
|
return 0;
|
2012-03-04 12:17:15 +07:00
|
|
|
if (uid_eq(inode->i_uid, fsuid))
|
2005-04-17 05:20:36 +07:00
|
|
|
return 0;
|
2012-03-04 12:17:15 +07:00
|
|
|
if (uid_eq(dir->i_uid, fsuid))
|
2005-04-17 05:20:36 +07:00
|
|
|
return 0;
|
2011-11-15 07:24:06 +07:00
|
|
|
return !inode_capable(inode, CAP_FOWNER);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Check whether we can remove a link victim from directory dir, check
|
|
|
|
* whether the type of victim is right.
|
|
|
|
* 1. We can't do it if dir is read-only (done in permission())
|
|
|
|
* 2. We should have write and exec permissions on dir
|
|
|
|
* 3. We can't remove anything from append-only dir
|
|
|
|
* 4. We can't do anything with immutable dir (done in permission())
|
|
|
|
* 5. If the sticky bit on dir is set we should either
|
|
|
|
* a. be owner of dir, or
|
|
|
|
* b. be owner of victim, or
|
|
|
|
* c. have CAP_FOWNER capability
|
|
|
|
* 6. If the victim is append-only or immutable we can't do antyhing with
|
|
|
|
* links pointing to it.
|
|
|
|
* 7. If we were asked to remove a directory and victim isn't one - ENOTDIR.
|
|
|
|
* 8. If we were asked to remove a non-directory and victim isn't one - EISDIR.
|
|
|
|
* 9. We can't remove a root or mountpoint.
|
|
|
|
* 10. We don't allow removal of NFS sillyrenamed files; it's handled by
|
|
|
|
* nfs_async_unlink().
|
|
|
|
*/
|
2006-01-15 04:20:43 +07:00
|
|
|
static int may_delete(struct inode *dir,struct dentry *victim,int isdir)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
int error;
|
|
|
|
|
|
|
|
if (!victim->d_inode)
|
|
|
|
return -ENOENT;
|
|
|
|
|
|
|
|
BUG_ON(victim->d_parent->d_inode != dir);
|
2009-12-25 17:07:33 +07:00
|
|
|
audit_inode_child(victim, dir);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2008-07-22 11:07:17 +07:00
|
|
|
error = inode_permission(dir, MAY_WRITE | MAY_EXEC);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
if (IS_APPEND(dir))
|
|
|
|
return -EPERM;
|
|
|
|
if (check_sticky(dir, victim->d_inode)||IS_APPEND(victim->d_inode)||
|
2008-11-20 06:36:38 +07:00
|
|
|
IS_IMMUTABLE(victim->d_inode) || IS_SWAPFILE(victim->d_inode))
|
2005-04-17 05:20:36 +07:00
|
|
|
return -EPERM;
|
|
|
|
if (isdir) {
|
|
|
|
if (!S_ISDIR(victim->d_inode->i_mode))
|
|
|
|
return -ENOTDIR;
|
|
|
|
if (IS_ROOT(victim))
|
|
|
|
return -EBUSY;
|
|
|
|
} else if (S_ISDIR(victim->d_inode->i_mode))
|
|
|
|
return -EISDIR;
|
|
|
|
if (IS_DEADDIR(dir))
|
|
|
|
return -ENOENT;
|
|
|
|
if (victim->d_flags & DCACHE_NFSFS_RENAMED)
|
|
|
|
return -EBUSY;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Check whether we can create an object with dentry child in directory
|
|
|
|
* dir.
|
|
|
|
* 1. We can't do it if child already exists (open has special treatment for
|
|
|
|
* this case, but since we are inlined it's OK)
|
|
|
|
* 2. We can't do it if dir is read-only (done in permission())
|
|
|
|
* 3. We should have write and exec permissions on dir
|
|
|
|
* 4. We can't do it if dir is immutable (done in permission())
|
|
|
|
*/
|
2008-07-30 20:08:48 +07:00
|
|
|
static inline int may_create(struct inode *dir, struct dentry *child)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
if (child->d_inode)
|
|
|
|
return -EEXIST;
|
|
|
|
if (IS_DEADDIR(dir))
|
|
|
|
return -ENOENT;
|
2008-07-22 11:07:17 +07:00
|
|
|
return inode_permission(dir, MAY_WRITE | MAY_EXEC);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* p1 and p2 should be directories on the same fs.
|
|
|
|
*/
|
|
|
|
struct dentry *lock_rename(struct dentry *p1, struct dentry *p2)
|
|
|
|
{
|
|
|
|
struct dentry *p;
|
|
|
|
|
|
|
|
if (p1 == p2) {
|
2006-07-03 14:25:05 +07:00
|
|
|
mutex_lock_nested(&p1->d_inode->i_mutex, I_MUTEX_PARENT);
|
2005-04-17 05:20:36 +07:00
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2006-03-23 18:00:33 +07:00
|
|
|
mutex_lock(&p1->d_inode->i_sb->s_vfs_rename_mutex);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2008-10-16 05:50:28 +07:00
|
|
|
p = d_ancestor(p2, p1);
|
|
|
|
if (p) {
|
|
|
|
mutex_lock_nested(&p2->d_inode->i_mutex, I_MUTEX_PARENT);
|
|
|
|
mutex_lock_nested(&p1->d_inode->i_mutex, I_MUTEX_CHILD);
|
|
|
|
return p;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2008-10-16 05:50:28 +07:00
|
|
|
p = d_ancestor(p1, p2);
|
|
|
|
if (p) {
|
|
|
|
mutex_lock_nested(&p1->d_inode->i_mutex, I_MUTEX_PARENT);
|
|
|
|
mutex_lock_nested(&p2->d_inode->i_mutex, I_MUTEX_CHILD);
|
|
|
|
return p;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2006-07-03 14:25:05 +07:00
|
|
|
mutex_lock_nested(&p1->d_inode->i_mutex, I_MUTEX_PARENT);
|
|
|
|
mutex_lock_nested(&p2->d_inode->i_mutex, I_MUTEX_CHILD);
|
2005-04-17 05:20:36 +07:00
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
void unlock_rename(struct dentry *p1, struct dentry *p2)
|
|
|
|
{
|
2006-01-10 06:59:24 +07:00
|
|
|
mutex_unlock(&p1->d_inode->i_mutex);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (p1 != p2) {
|
2006-01-10 06:59:24 +07:00
|
|
|
mutex_unlock(&p2->d_inode->i_mutex);
|
2006-03-23 18:00:33 +07:00
|
|
|
mutex_unlock(&p1->d_inode->i_sb->s_vfs_rename_mutex);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2011-07-26 12:42:34 +07:00
|
|
|
int vfs_create(struct inode *dir, struct dentry *dentry, umode_t mode,
|
2005-04-17 05:20:36 +07:00
|
|
|
struct nameidata *nd)
|
|
|
|
{
|
2008-07-30 20:08:48 +07:00
|
|
|
int error = may_create(dir, dentry);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
|
2008-12-04 22:06:33 +07:00
|
|
|
if (!dir->i_op->create)
|
2005-04-17 05:20:36 +07:00
|
|
|
return -EACCES; /* shouldn't it be ENOSYS? */
|
|
|
|
mode &= S_IALLUGO;
|
|
|
|
mode |= S_IFREG;
|
|
|
|
error = security_inode_create(dir, dentry, mode);
|
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
error = dir->i_op->create(dir, dentry, mode, nd);
|
2005-09-10 03:01:44 +07:00
|
|
|
if (!error)
|
2005-11-03 22:57:06 +07:00
|
|
|
fsnotify_create(dir, dentry);
|
2005-04-17 05:20:36 +07:00
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2011-03-12 00:08:24 +07:00
|
|
|
static int may_open(struct path *path, int acc_mode, int flag)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2008-10-24 14:58:10 +07:00
|
|
|
struct dentry *dentry = path->dentry;
|
2005-04-17 05:20:36 +07:00
|
|
|
struct inode *inode = dentry->d_inode;
|
|
|
|
int error;
|
|
|
|
|
2011-03-14 03:42:14 +07:00
|
|
|
/* O_PATH? */
|
|
|
|
if (!acc_mode)
|
|
|
|
return 0;
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
if (!inode)
|
|
|
|
return -ENOENT;
|
|
|
|
|
2009-01-06 01:27:23 +07:00
|
|
|
switch (inode->i_mode & S_IFMT) {
|
|
|
|
case S_IFLNK:
|
2005-04-17 05:20:36 +07:00
|
|
|
return -ELOOP;
|
2009-01-06 01:27:23 +07:00
|
|
|
case S_IFDIR:
|
|
|
|
if (acc_mode & MAY_WRITE)
|
|
|
|
return -EISDIR;
|
|
|
|
break;
|
|
|
|
case S_IFBLK:
|
|
|
|
case S_IFCHR:
|
2008-10-24 14:58:10 +07:00
|
|
|
if (path->mnt->mnt_flags & MNT_NODEV)
|
2005-04-17 05:20:36 +07:00
|
|
|
return -EACCES;
|
2009-01-06 01:27:23 +07:00
|
|
|
/*FALLTHRU*/
|
|
|
|
case S_IFIFO:
|
|
|
|
case S_IFSOCK:
|
2005-04-17 05:20:36 +07:00
|
|
|
flag &= ~O_TRUNC;
|
2009-01-06 01:27:23 +07:00
|
|
|
break;
|
2008-02-16 05:37:48 +07:00
|
|
|
}
|
2007-10-17 13:31:14 +07:00
|
|
|
|
2008-10-24 14:58:10 +07:00
|
|
|
error = inode_permission(inode, acc_mode);
|
2007-10-17 13:31:14 +07:00
|
|
|
if (error)
|
|
|
|
return error;
|
2009-02-04 21:06:57 +07:00
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* An append-only file must be opened in append mode for writing.
|
|
|
|
*/
|
|
|
|
if (IS_APPEND(inode)) {
|
2009-12-24 18:47:55 +07:00
|
|
|
if ((flag & O_ACCMODE) != O_RDONLY && !(flag & O_APPEND))
|
2009-12-16 15:54:00 +07:00
|
|
|
return -EPERM;
|
2005-04-17 05:20:36 +07:00
|
|
|
if (flag & O_TRUNC)
|
2009-12-16 15:54:00 +07:00
|
|
|
return -EPERM;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/* O_NOATIME can only be set by the owner or superuser */
|
2011-03-24 06:43:26 +07:00
|
|
|
if (flag & O_NOATIME && !inode_owner_or_capable(inode))
|
2009-12-16 15:54:00 +07:00
|
|
|
return -EPERM;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2011-09-21 21:58:13 +07:00
|
|
|
return 0;
|
2009-12-16 15:54:00 +07:00
|
|
|
}
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2010-12-08 04:19:50 +07:00
|
|
|
static int handle_truncate(struct file *filp)
|
2009-12-16 15:54:00 +07:00
|
|
|
{
|
2010-12-08 04:19:50 +07:00
|
|
|
struct path *path = &filp->f_path;
|
2009-12-16 15:54:00 +07:00
|
|
|
struct inode *inode = path->dentry->d_inode;
|
|
|
|
int error = get_write_access(inode);
|
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
/*
|
|
|
|
* Refuse to truncate files with mandatory locks held on them.
|
|
|
|
*/
|
|
|
|
error = locks_verify_locked(inode);
|
|
|
|
if (!error)
|
2010-06-02 11:24:43 +07:00
|
|
|
error = security_path_truncate(path);
|
2009-12-16 15:54:00 +07:00
|
|
|
if (!error) {
|
|
|
|
error = do_truncate(path->dentry, 0,
|
|
|
|
ATTR_MTIME|ATTR_CTIME|ATTR_OPEN,
|
2010-12-08 04:19:50 +07:00
|
|
|
filp);
|
2009-12-16 15:54:00 +07:00
|
|
|
}
|
|
|
|
put_write_access(inode);
|
2009-09-05 00:08:46 +07:00
|
|
|
return error;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2008-02-16 05:37:27 +07:00
|
|
|
static inline int open_to_namei_flags(int flag)
|
|
|
|
{
|
2011-06-26 06:15:54 +07:00
|
|
|
if ((flag & O_ACCMODE) == 3)
|
|
|
|
flag--;
|
2008-02-16 05:37:27 +07:00
|
|
|
return flag;
|
|
|
|
}
|
|
|
|
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
/*
|
2011-03-06 10:58:25 +07:00
|
|
|
* Handle the last step of open()
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
*/
|
2009-12-24 13:58:28 +07:00
|
|
|
static struct file *do_last(struct nameidata *nd, struct path *path,
|
2011-02-24 01:39:45 +07:00
|
|
|
const struct open_flags *op, const char *pathname)
|
2009-12-24 13:58:28 +07:00
|
|
|
{
|
2009-12-24 14:12:06 +07:00
|
|
|
struct dentry *dir = nd->path.dentry;
|
2011-03-09 12:59:59 +07:00
|
|
|
struct dentry *dentry;
|
2011-03-09 12:36:45 +07:00
|
|
|
int open_flag = op->open_flag;
|
2011-03-09 12:59:59 +07:00
|
|
|
int will_truncate = open_flag & O_TRUNC;
|
2011-03-09 12:36:45 +07:00
|
|
|
int want_write = 0;
|
2011-03-14 03:42:14 +07:00
|
|
|
int acc_mode = op->acc_mode;
|
2009-12-24 13:58:28 +07:00
|
|
|
struct file *filp;
|
2012-05-21 22:30:07 +07:00
|
|
|
struct inode *inode;
|
2012-05-21 22:30:09 +07:00
|
|
|
int symlink_ok = 0;
|
2012-05-21 22:30:19 +07:00
|
|
|
struct path save_parent = { .dentry = NULL, .mnt = NULL };
|
|
|
|
bool retried = false;
|
2011-02-23 03:50:10 +07:00
|
|
|
int error;
|
2009-12-26 22:56:19 +07:00
|
|
|
|
2011-02-24 01:39:45 +07:00
|
|
|
nd->flags &= ~LOOKUP_PARENT;
|
|
|
|
nd->flags |= op->intent;
|
|
|
|
|
2009-12-26 22:56:19 +07:00
|
|
|
switch (nd->last_type) {
|
|
|
|
case LAST_DOTDOT:
|
2010-05-24 13:57:56 +07:00
|
|
|
case LAST_DOT:
|
2011-03-06 10:58:25 +07:00
|
|
|
error = handle_dots(nd, nd->last_type);
|
|
|
|
if (error)
|
|
|
|
return ERR_PTR(error);
|
2009-12-26 22:56:19 +07:00
|
|
|
/* fallthrough */
|
|
|
|
case LAST_ROOT:
|
2011-03-25 22:00:12 +07:00
|
|
|
error = complete_walk(nd);
|
2011-02-23 03:50:10 +07:00
|
|
|
if (error)
|
2011-03-25 22:00:12 +07:00
|
|
|
return ERR_PTR(error);
|
2011-03-06 10:58:25 +07:00
|
|
|
audit_inode(pathname, nd->path.dentry);
|
2011-03-09 12:36:45 +07:00
|
|
|
if (open_flag & O_CREAT) {
|
2011-03-06 10:58:25 +07:00
|
|
|
error = -EISDIR;
|
|
|
|
goto exit;
|
|
|
|
}
|
|
|
|
goto ok;
|
2009-12-26 22:56:19 +07:00
|
|
|
case LAST_BIND:
|
2011-03-25 22:00:12 +07:00
|
|
|
error = complete_walk(nd);
|
2011-02-23 03:50:10 +07:00
|
|
|
if (error)
|
2011-03-25 22:00:12 +07:00
|
|
|
return ERR_PTR(error);
|
2009-12-26 22:56:19 +07:00
|
|
|
audit_inode(pathname, dir);
|
2009-12-26 19:01:01 +07:00
|
|
|
goto ok;
|
2009-12-26 22:56:19 +07:00
|
|
|
}
|
2009-12-26 19:01:01 +07:00
|
|
|
|
2011-03-09 12:36:45 +07:00
|
|
|
if (!(open_flag & O_CREAT)) {
|
2011-03-06 10:58:25 +07:00
|
|
|
if (nd->last.name[nd->last.len])
|
|
|
|
nd->flags |= LOOKUP_FOLLOW | LOOKUP_DIRECTORY;
|
2011-03-14 03:42:14 +07:00
|
|
|
if (open_flag & O_PATH && !(nd->flags & LOOKUP_FOLLOW))
|
|
|
|
symlink_ok = 1;
|
2011-03-06 10:58:25 +07:00
|
|
|
/* we _can_ be in RCU mode here */
|
2012-05-21 22:30:07 +07:00
|
|
|
error = lookup_fast(nd, &nd->last, path, &inode);
|
|
|
|
if (unlikely(error)) {
|
|
|
|
if (error < 0)
|
|
|
|
goto exit;
|
|
|
|
|
|
|
|
error = lookup_slow(nd, &nd->last, path);
|
|
|
|
if (error < 0)
|
|
|
|
goto exit;
|
|
|
|
|
|
|
|
inode = path->dentry->d_inode;
|
|
|
|
}
|
2012-05-21 22:30:14 +07:00
|
|
|
goto finish_lookup;
|
2011-03-06 10:58:25 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/* create side of things */
|
2011-11-08 04:21:26 +07:00
|
|
|
/*
|
|
|
|
* This will *only* deal with leaving RCU mode - LOOKUP_JUMPED has been
|
|
|
|
* cleared when we got to the last component we are about to look up
|
|
|
|
*/
|
2011-03-25 22:00:12 +07:00
|
|
|
error = complete_walk(nd);
|
|
|
|
if (error)
|
|
|
|
return ERR_PTR(error);
|
2011-03-06 10:58:25 +07:00
|
|
|
|
|
|
|
audit_inode(pathname, dir);
|
2011-02-23 03:50:10 +07:00
|
|
|
error = -EISDIR;
|
2009-12-26 22:56:19 +07:00
|
|
|
/* trailing slashes? */
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
if (nd->last.name[nd->last.len])
|
|
|
|
goto exit;
|
2009-12-24 15:39:50 +07:00
|
|
|
|
2012-05-21 22:30:19 +07:00
|
|
|
retry_lookup:
|
2009-12-24 14:12:06 +07:00
|
|
|
mutex_lock(&dir->d_inode->i_mutex);
|
|
|
|
|
2011-03-09 12:59:59 +07:00
|
|
|
dentry = lookup_hash(nd);
|
|
|
|
error = PTR_ERR(dentry);
|
|
|
|
if (IS_ERR(dentry)) {
|
2009-12-24 13:58:28 +07:00
|
|
|
mutex_unlock(&dir->d_inode->i_mutex);
|
|
|
|
goto exit;
|
|
|
|
}
|
|
|
|
|
2011-03-09 12:59:59 +07:00
|
|
|
path->dentry = dentry;
|
|
|
|
path->mnt = nd->path.mnt;
|
|
|
|
|
2009-12-24 13:58:28 +07:00
|
|
|
/* Negative dentry, just create the file */
|
2011-03-09 12:59:59 +07:00
|
|
|
if (!dentry->d_inode) {
|
2011-11-22 02:59:34 +07:00
|
|
|
umode_t mode = op->mode;
|
2011-03-09 12:59:59 +07:00
|
|
|
if (!IS_POSIXACL(dir->d_inode))
|
|
|
|
mode &= ~current_umask();
|
2009-12-24 13:58:28 +07:00
|
|
|
/*
|
|
|
|
* This write is needed to ensure that a
|
2011-03-09 12:59:59 +07:00
|
|
|
* rw->ro transition does not occur between
|
2009-12-24 13:58:28 +07:00
|
|
|
* the time when the file is created and when
|
|
|
|
* a permanent write count is taken through
|
|
|
|
* the 'struct file' in nameidata_to_filp().
|
|
|
|
*/
|
|
|
|
error = mnt_want_write(nd->path.mnt);
|
|
|
|
if (error)
|
|
|
|
goto exit_mutex_unlock;
|
2011-03-09 12:36:45 +07:00
|
|
|
want_write = 1;
|
2011-03-09 12:17:27 +07:00
|
|
|
/* Don't check for write permission, don't truncate */
|
2011-03-09 12:36:45 +07:00
|
|
|
open_flag &= ~O_TRUNC;
|
2011-03-09 12:59:59 +07:00
|
|
|
will_truncate = 0;
|
2011-03-14 03:42:14 +07:00
|
|
|
acc_mode = MAY_OPEN;
|
2011-03-09 12:59:59 +07:00
|
|
|
error = security_path_mknod(&nd->path, dentry, mode, 0);
|
|
|
|
if (error)
|
|
|
|
goto exit_mutex_unlock;
|
|
|
|
error = vfs_create(dir->d_inode, dentry, mode, nd);
|
|
|
|
if (error)
|
|
|
|
goto exit_mutex_unlock;
|
|
|
|
mutex_unlock(&dir->d_inode->i_mutex);
|
|
|
|
dput(nd->path.dentry);
|
|
|
|
nd->path.dentry = dentry;
|
2011-03-09 12:36:45 +07:00
|
|
|
goto common;
|
2009-12-24 13:58:28 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* It already exists.
|
|
|
|
*/
|
|
|
|
mutex_unlock(&dir->d_inode->i_mutex);
|
|
|
|
audit_inode(pathname, path->dentry);
|
|
|
|
|
|
|
|
error = -EEXIST;
|
2011-03-09 12:36:45 +07:00
|
|
|
if (open_flag & O_EXCL)
|
2009-12-24 13:58:28 +07:00
|
|
|
goto exit_dput;
|
|
|
|
|
Add a dentry op to handle automounting rather than abusing follow_link()
Add a dentry op (d_automount) to handle automounting directories rather than
abusing the follow_link() inode operation. The operation is keyed off a new
dentry flag (DCACHE_NEED_AUTOMOUNT).
This also makes it easier to add an AT_ flag to suppress terminal segment
automount during pathwalk and removes the need for the kludge code in the
pathwalk algorithm to handle directories with follow_link() semantics.
The ->d_automount() dentry operation:
struct vfsmount *(*d_automount)(struct path *mountpoint);
takes a pointer to the directory to be mounted upon, which is expected to
provide sufficient data to determine what should be mounted. If successful, it
should return the vfsmount struct it creates (which it should also have added
to the namespace using do_add_mount() or similar). If there's a collision with
another automount attempt, NULL should be returned. If the directory specified
by the parameter should be used directly rather than being mounted upon,
-EISDIR should be returned. In any other case, an error code should be
returned.
The ->d_automount() operation is called with no locks held and may sleep. At
this point the pathwalk algorithm will be in ref-walk mode.
Within fs/namei.c itself, a new pathwalk subroutine (follow_automount()) is
added to handle mountpoints. It will return -EREMOTE if the automount flag was
set, but no d_automount() op was supplied, -ELOOP if we've encountered too many
symlinks or mountpoints, -EISDIR if the walk point should be used without
mounting and 0 if successful. The path will be updated to point to the mounted
filesystem if a successful automount took place.
__follow_mount() is replaced by follow_managed() which is more generic
(especially with the patch that adds ->d_manage()). This handles transits from
directories during pathwalk, including automounting and skipping over
mountpoints (and holding processes with the next patch).
__follow_mount_rcu() will jump out of RCU-walk mode if it encounters an
automount point with nothing mounted on it.
follow_dotdot*() does not handle automounts as you don't want to trigger them
whilst following "..".
I've also extracted the mount/don't-mount logic from autofs4 and included it
here. It makes the mount go ahead anyway if someone calls open() or creat(),
tries to traverse the directory, tries to chdir/chroot/etc. into the directory,
or sticks a '/' on the end of the pathname. If they do a stat(), however,
they'll only trigger the automount if they didn't also say O_NOFOLLOW.
I've also added an inode flag (S_AUTOMOUNT) so that filesystems can mark their
inodes as automount points. This flag is automatically propagated to the
dentry as DCACHE_NEED_AUTOMOUNT by __d_instantiate(). This saves NFS and could
save AFS a private flag bit apiece, but is not strictly necessary. It would be
preferable to do the propagation in d_set_d_op(), but that doesn't normally
have access to the inode.
[AV: fixed breakage in case if __follow_mount_rcu() fails and nameidata_drop_rcu()
succeeds in RCU case of do_lookup(); we need to fall through to non-RCU case after
that, rather than just returning with ungrabbed *path]
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:21 +07:00
|
|
|
error = follow_managed(path, nd->flags);
|
|
|
|
if (error < 0)
|
|
|
|
goto exit_dput;
|
2009-12-24 13:58:28 +07:00
|
|
|
|
2011-11-08 04:21:26 +07:00
|
|
|
if (error)
|
|
|
|
nd->flags |= LOOKUP_JUMPED;
|
|
|
|
|
2012-05-21 22:30:08 +07:00
|
|
|
BUG_ON(nd->flags & LOOKUP_RCU);
|
|
|
|
inode = path->dentry->d_inode;
|
2012-05-21 22:30:14 +07:00
|
|
|
finish_lookup:
|
|
|
|
/* we _can_ be in RCU mode here */
|
2009-12-24 13:58:28 +07:00
|
|
|
error = -ENOENT;
|
2012-05-21 22:30:10 +07:00
|
|
|
if (!inode) {
|
|
|
|
path_to_nameidata(path, nd);
|
|
|
|
goto exit;
|
|
|
|
}
|
2009-12-26 19:04:50 +07:00
|
|
|
|
2012-05-21 22:30:09 +07:00
|
|
|
if (should_follow_link(inode, !symlink_ok)) {
|
|
|
|
if (nd->flags & LOOKUP_RCU) {
|
|
|
|
if (unlikely(unlazy_walk(nd, path->dentry))) {
|
|
|
|
error = -ECHILD;
|
|
|
|
goto exit;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
BUG_ON(inode != path->dentry->d_inode);
|
2009-12-24 13:58:28 +07:00
|
|
|
return NULL;
|
2012-05-21 22:30:09 +07:00
|
|
|
}
|
2009-12-24 13:58:28 +07:00
|
|
|
|
2012-05-21 22:30:19 +07:00
|
|
|
if ((nd->flags & LOOKUP_RCU) || nd->path.mnt != path->mnt) {
|
|
|
|
path_to_nameidata(path, nd);
|
|
|
|
} else {
|
|
|
|
save_parent.dentry = nd->path.dentry;
|
|
|
|
save_parent.mnt = mntget(path->mnt);
|
|
|
|
nd->path.dentry = path->dentry;
|
|
|
|
|
|
|
|
}
|
2012-05-21 22:30:08 +07:00
|
|
|
nd->inode = inode;
|
2011-11-08 04:21:26 +07:00
|
|
|
/* Why this, you ask? _Now_ we might have grown LOOKUP_JUMPED... */
|
|
|
|
error = complete_walk(nd);
|
2012-05-21 22:30:19 +07:00
|
|
|
if (error) {
|
|
|
|
path_put(&save_parent);
|
2012-03-06 19:56:33 +07:00
|
|
|
return ERR_PTR(error);
|
2012-05-21 22:30:19 +07:00
|
|
|
}
|
2009-12-24 13:58:28 +07:00
|
|
|
error = -EISDIR;
|
2012-05-21 22:30:12 +07:00
|
|
|
if ((open_flag & O_CREAT) && S_ISDIR(nd->inode->i_mode))
|
2009-12-24 13:58:28 +07:00
|
|
|
goto exit;
|
2012-05-21 22:30:11 +07:00
|
|
|
error = -ENOTDIR;
|
|
|
|
if ((nd->flags & LOOKUP_DIRECTORY) && !nd->inode->i_op->lookup)
|
|
|
|
goto exit;
|
2012-05-21 22:30:13 +07:00
|
|
|
audit_inode(pathname, nd->path.dentry);
|
2009-12-26 19:01:01 +07:00
|
|
|
ok:
|
2011-03-09 12:59:59 +07:00
|
|
|
if (!S_ISREG(nd->inode->i_mode))
|
|
|
|
will_truncate = 0;
|
|
|
|
|
2011-03-09 12:13:14 +07:00
|
|
|
if (will_truncate) {
|
|
|
|
error = mnt_want_write(nd->path.mnt);
|
|
|
|
if (error)
|
|
|
|
goto exit;
|
2011-03-09 12:36:45 +07:00
|
|
|
want_write = 1;
|
2011-03-09 12:13:14 +07:00
|
|
|
}
|
2011-03-09 12:36:45 +07:00
|
|
|
common:
|
2011-03-14 03:42:14 +07:00
|
|
|
error = may_open(&nd->path, acc_mode, open_flag);
|
2011-03-09 12:36:45 +07:00
|
|
|
if (error)
|
2011-03-09 12:13:14 +07:00
|
|
|
goto exit;
|
|
|
|
filp = nameidata_to_filp(nd);
|
2012-05-21 22:30:19 +07:00
|
|
|
if (filp == ERR_PTR(-EOPENSTALE) && save_parent.dentry && !retried) {
|
|
|
|
BUG_ON(save_parent.dentry != dir);
|
|
|
|
path_put(&nd->path);
|
|
|
|
nd->path = save_parent;
|
|
|
|
nd->inode = dir->d_inode;
|
|
|
|
save_parent.mnt = NULL;
|
|
|
|
save_parent.dentry = NULL;
|
|
|
|
if (want_write) {
|
|
|
|
mnt_drop_write(nd->path.mnt);
|
|
|
|
want_write = 0;
|
|
|
|
}
|
|
|
|
retried = true;
|
|
|
|
goto retry_lookup;
|
|
|
|
}
|
2011-03-09 12:13:14 +07:00
|
|
|
if (!IS_ERR(filp)) {
|
|
|
|
error = ima_file_check(filp, op->acc_mode);
|
|
|
|
if (error) {
|
|
|
|
fput(filp);
|
|
|
|
filp = ERR_PTR(error);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
if (!IS_ERR(filp)) {
|
|
|
|
if (will_truncate) {
|
|
|
|
error = handle_truncate(filp);
|
|
|
|
if (error) {
|
|
|
|
fput(filp);
|
|
|
|
filp = ERR_PTR(error);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
2011-03-09 12:36:45 +07:00
|
|
|
out:
|
|
|
|
if (want_write)
|
2011-03-09 12:13:14 +07:00
|
|
|
mnt_drop_write(nd->path.mnt);
|
2012-05-21 22:30:19 +07:00
|
|
|
path_put(&save_parent);
|
2012-05-21 22:30:06 +07:00
|
|
|
terminate_walk(nd);
|
2009-12-24 13:58:28 +07:00
|
|
|
return filp;
|
|
|
|
|
|
|
|
exit_mutex_unlock:
|
|
|
|
mutex_unlock(&dir->d_inode->i_mutex);
|
|
|
|
exit_dput:
|
|
|
|
path_put_conditional(path, nd);
|
|
|
|
exit:
|
2011-03-09 12:36:45 +07:00
|
|
|
filp = ERR_PTR(error);
|
|
|
|
goto out;
|
2009-12-24 13:58:28 +07:00
|
|
|
}
|
|
|
|
|
2011-02-24 05:54:08 +07:00
|
|
|
static struct file *path_openat(int dfd, const char *pathname,
|
2011-03-12 00:08:24 +07:00
|
|
|
struct nameidata *nd, const struct open_flags *op, int flags)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2011-03-06 10:58:25 +07:00
|
|
|
struct file *base = NULL;
|
2008-02-16 05:37:48 +07:00
|
|
|
struct file *filp;
|
2010-01-14 03:01:15 +07:00
|
|
|
struct path path;
|
2011-02-24 05:54:08 +07:00
|
|
|
int error;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
|
|
|
|
filp = get_empty_filp();
|
|
|
|
if (!filp)
|
|
|
|
return ERR_PTR(-ENFILE);
|
|
|
|
|
2011-02-24 05:44:09 +07:00
|
|
|
filp->f_flags = op->open_flag;
|
2011-03-12 00:08:24 +07:00
|
|
|
nd->intent.open.file = filp;
|
|
|
|
nd->intent.open.flags = open_to_namei_flags(op->open_flag);
|
|
|
|
nd->intent.open.create_mode = op->mode;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
|
2011-03-12 00:08:24 +07:00
|
|
|
error = path_init(dfd, pathname, flags | LOOKUP_PARENT, nd, &base);
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
if (unlikely(error))
|
2011-02-24 05:54:08 +07:00
|
|
|
goto out_filp;
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
|
2011-03-06 10:58:25 +07:00
|
|
|
current->total_link_count = 0;
|
2011-03-12 00:08:24 +07:00
|
|
|
error = link_path_walk(pathname, nd);
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
if (unlikely(error))
|
|
|
|
goto out_filp;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2011-03-12 00:08:24 +07:00
|
|
|
filp = do_last(nd, &path, op, pathname);
|
2009-12-26 19:16:40 +07:00
|
|
|
while (unlikely(!filp)) { /* trailing symlink */
|
2011-01-14 15:42:43 +07:00
|
|
|
struct path link = path;
|
2009-12-26 20:37:05 +07:00
|
|
|
void *cookie;
|
2011-03-15 09:20:34 +07:00
|
|
|
if (!(nd->flags & LOOKUP_FOLLOW)) {
|
2011-03-12 00:08:24 +07:00
|
|
|
path_put_conditional(&path, nd);
|
|
|
|
path_put(&nd->path);
|
2011-03-10 04:22:18 +07:00
|
|
|
filp = ERR_PTR(-ELOOP);
|
|
|
|
break;
|
|
|
|
}
|
2011-03-12 00:08:24 +07:00
|
|
|
nd->flags |= LOOKUP_PARENT;
|
|
|
|
nd->flags &= ~(LOOKUP_OPEN|LOOKUP_CREATE|LOOKUP_EXCL);
|
2011-03-15 09:20:34 +07:00
|
|
|
error = follow_link(&link, nd, &cookie);
|
2011-02-24 01:39:45 +07:00
|
|
|
if (unlikely(error))
|
2011-02-23 10:27:28 +07:00
|
|
|
filp = ERR_PTR(error);
|
2011-02-24 01:39:45 +07:00
|
|
|
else
|
2011-03-12 00:08:24 +07:00
|
|
|
filp = do_last(nd, &path, op, pathname);
|
2011-03-15 09:20:34 +07:00
|
|
|
put_link(nd, &link, cookie);
|
2009-12-26 19:16:40 +07:00
|
|
|
}
|
2009-12-26 19:09:49 +07:00
|
|
|
out:
|
2011-03-12 00:08:24 +07:00
|
|
|
if (nd->root.mnt && !(nd->flags & LOOKUP_ROOT))
|
|
|
|
path_put(&nd->root);
|
2011-03-06 10:58:25 +07:00
|
|
|
if (base)
|
|
|
|
fput(base);
|
2011-03-12 00:08:24 +07:00
|
|
|
release_open_intent(nd);
|
2012-05-21 22:30:19 +07:00
|
|
|
if (filp == ERR_PTR(-EOPENSTALE)) {
|
|
|
|
if (flags & LOOKUP_RCU)
|
|
|
|
filp = ERR_PTR(-ECHILD);
|
|
|
|
else
|
|
|
|
filp = ERR_PTR(-ESTALE);
|
|
|
|
}
|
2009-12-26 19:09:49 +07:00
|
|
|
return filp;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
fs: rcu-walk for path lookup
Perform common cases of path lookups without any stores or locking in the
ancestor dentry elements. This is called rcu-walk, as opposed to the current
algorithm which is a refcount based walk, or ref-walk.
This results in far fewer atomic operations on every path element,
significantly improving path lookup performance. It also avoids cacheline
bouncing on common dentries, significantly improving scalability.
The overall design is like this:
* LOOKUP_RCU is set in nd->flags, which distinguishes rcu-walk from ref-walk.
* Take the RCU lock for the entire path walk, starting with the acquiring
of the starting path (eg. root/cwd/fd-path). So now dentry refcounts are
not required for dentry persistence.
* synchronize_rcu is called when unregistering a filesystem, so we can
access d_ops and i_ops during rcu-walk.
* Similarly take the vfsmount lock for the entire path walk. So now mnt
refcounts are not required for persistence. Also we are free to perform mount
lookups, and to assume dentry mount points and mount roots are stable up and
down the path.
* Have a per-dentry seqlock to protect the dentry name, parent, and inode,
so we can load this tuple atomically, and also check whether any of its
members have changed.
* Dentry lookups (based on parent, candidate string tuple) recheck the parent
sequence after the child is found in case anything changed in the parent
during the path walk.
* inode is also RCU protected so we can load d_inode and use the inode for
limited things.
* i_mode, i_uid, i_gid can be tested for exec permissions during path walk.
* i_op can be loaded.
When we reach the destination dentry, we lock it, recheck lookup sequence,
and increment its refcount and mountpoint refcount. RCU and vfsmount locks
are dropped. This is termed "dropping rcu-walk". If the dentry refcount does
not match, we can not drop rcu-walk gracefully at the current point in the
lokup, so instead return -ECHILD (for want of a better errno). This signals the
path walking code to re-do the entire lookup with a ref-walk.
Aside from the final dentry, there are other situations that may be encounted
where we cannot continue rcu-walk. In that case, we drop rcu-walk (ie. take
a reference on the last good dentry) and continue with a ref-walk. Again, if
we can drop rcu-walk gracefully, we return -ECHILD and do the whole lookup
using ref-walk. But it is very important that we can continue with ref-walk
for most cases, particularly to avoid the overhead of double lookups, and to
gain the scalability advantages on common path elements (like cwd and root).
The cases where rcu-walk cannot continue are:
* NULL dentry (ie. any uncached path element)
* parent with d_inode->i_op->permission or ACLs
* dentries with d_revalidate
* Following links
In future patches, permission checks and d_revalidate become rcu-walk aware. It
may be possible eventually to make following links rcu-walk aware.
Uncached path elements will always require dropping to ref-walk mode, at the
very least because i_mutex needs to be grabbed, and objects allocated.
Signed-off-by: Nick Piggin <npiggin@kernel.dk>
2011-01-07 13:49:52 +07:00
|
|
|
out_filp:
|
2009-12-26 19:16:40 +07:00
|
|
|
filp = ERR_PTR(error);
|
2009-12-26 19:09:49 +07:00
|
|
|
goto out;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2011-02-24 05:54:08 +07:00
|
|
|
struct file *do_filp_open(int dfd, const char *pathname,
|
|
|
|
const struct open_flags *op, int flags)
|
|
|
|
{
|
2011-03-12 00:08:24 +07:00
|
|
|
struct nameidata nd;
|
2011-02-24 05:54:08 +07:00
|
|
|
struct file *filp;
|
|
|
|
|
2011-03-12 00:08:24 +07:00
|
|
|
filp = path_openat(dfd, pathname, &nd, op, flags | LOOKUP_RCU);
|
2011-02-24 05:54:08 +07:00
|
|
|
if (unlikely(filp == ERR_PTR(-ECHILD)))
|
2011-03-12 00:08:24 +07:00
|
|
|
filp = path_openat(dfd, pathname, &nd, op, flags);
|
2011-02-24 05:54:08 +07:00
|
|
|
if (unlikely(filp == ERR_PTR(-ESTALE)))
|
2011-03-12 00:08:24 +07:00
|
|
|
filp = path_openat(dfd, pathname, &nd, op, flags | LOOKUP_REVAL);
|
2011-02-24 05:54:08 +07:00
|
|
|
return filp;
|
|
|
|
}
|
|
|
|
|
2011-03-12 00:08:24 +07:00
|
|
|
struct file *do_file_open_root(struct dentry *dentry, struct vfsmount *mnt,
|
|
|
|
const char *name, const struct open_flags *op, int flags)
|
|
|
|
{
|
|
|
|
struct nameidata nd;
|
|
|
|
struct file *file;
|
|
|
|
|
|
|
|
nd.root.mnt = mnt;
|
|
|
|
nd.root.dentry = dentry;
|
|
|
|
|
|
|
|
flags |= LOOKUP_ROOT;
|
|
|
|
|
2011-03-14 03:42:14 +07:00
|
|
|
if (dentry->d_inode->i_op->follow_link && op->intent & LOOKUP_OPEN)
|
2011-03-12 00:08:24 +07:00
|
|
|
return ERR_PTR(-ELOOP);
|
|
|
|
|
|
|
|
file = path_openat(-1, name, &nd, op, flags | LOOKUP_RCU);
|
|
|
|
if (unlikely(file == ERR_PTR(-ECHILD)))
|
|
|
|
file = path_openat(-1, name, &nd, op, flags);
|
|
|
|
if (unlikely(file == ERR_PTR(-ESTALE)))
|
|
|
|
file = path_openat(-1, name, &nd, op, flags | LOOKUP_REVAL);
|
|
|
|
return file;
|
|
|
|
}
|
|
|
|
|
2011-06-28 03:53:43 +07:00
|
|
|
struct dentry *kern_path_create(int dfd, const char *pathname, struct path *path, int is_dir)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2005-06-23 14:09:49 +07:00
|
|
|
struct dentry *dentry = ERR_PTR(-EEXIST);
|
2011-06-28 03:53:43 +07:00
|
|
|
struct nameidata nd;
|
|
|
|
int error = do_path_lookup(dfd, pathname, LOOKUP_PARENT, &nd);
|
|
|
|
if (error)
|
|
|
|
return ERR_PTR(error);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2005-06-23 14:09:49 +07:00
|
|
|
/*
|
|
|
|
* Yucky last component or no last component at all?
|
|
|
|
* (foo/., foo/.., /////)
|
|
|
|
*/
|
2011-06-28 03:53:43 +07:00
|
|
|
if (nd.last_type != LAST_NORM)
|
|
|
|
goto out;
|
|
|
|
nd.flags &= ~LOOKUP_PARENT;
|
|
|
|
nd.flags |= LOOKUP_CREATE | LOOKUP_EXCL;
|
|
|
|
nd.intent.open.flags = O_EXCL;
|
2005-06-23 14:09:49 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Do the final lookup.
|
|
|
|
*/
|
2011-06-28 03:53:43 +07:00
|
|
|
mutex_lock_nested(&nd.path.dentry->d_inode->i_mutex, I_MUTEX_PARENT);
|
|
|
|
dentry = lookup_hash(&nd);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (IS_ERR(dentry))
|
|
|
|
goto fail;
|
2005-06-23 14:09:49 +07:00
|
|
|
|
2008-05-15 15:49:12 +07:00
|
|
|
if (dentry->d_inode)
|
|
|
|
goto eexist;
|
2005-06-23 14:09:49 +07:00
|
|
|
/*
|
|
|
|
* Special case - lookup gave negative, but... we had foo/bar/
|
|
|
|
* From the vfs_mknod() POV we just have a negative dentry -
|
|
|
|
* all is fine. Let's be bastards - you had / on the end, you've
|
|
|
|
* been asking for (non-existent) directory. -ENOENT for you.
|
|
|
|
*/
|
2011-06-28 03:53:43 +07:00
|
|
|
if (unlikely(!is_dir && nd.last.name[nd.last.len])) {
|
2008-05-15 15:49:12 +07:00
|
|
|
dput(dentry);
|
|
|
|
dentry = ERR_PTR(-ENOENT);
|
2011-06-28 03:53:43 +07:00
|
|
|
goto fail;
|
2008-05-15 15:49:12 +07:00
|
|
|
}
|
2011-06-28 03:53:43 +07:00
|
|
|
*path = nd.path;
|
2005-04-17 05:20:36 +07:00
|
|
|
return dentry;
|
2008-05-15 15:49:12 +07:00
|
|
|
eexist:
|
2005-04-17 05:20:36 +07:00
|
|
|
dput(dentry);
|
2008-05-15 15:49:12 +07:00
|
|
|
dentry = ERR_PTR(-EEXIST);
|
2005-04-17 05:20:36 +07:00
|
|
|
fail:
|
2011-06-28 03:53:43 +07:00
|
|
|
mutex_unlock(&nd.path.dentry->d_inode->i_mutex);
|
|
|
|
out:
|
|
|
|
path_put(&nd.path);
|
2005-04-17 05:20:36 +07:00
|
|
|
return dentry;
|
|
|
|
}
|
2011-06-26 22:50:15 +07:00
|
|
|
EXPORT_SYMBOL(kern_path_create);
|
|
|
|
|
|
|
|
struct dentry *user_path_create(int dfd, const char __user *pathname, struct path *path, int is_dir)
|
|
|
|
{
|
|
|
|
char *tmp = getname(pathname);
|
|
|
|
struct dentry *res;
|
|
|
|
if (IS_ERR(tmp))
|
|
|
|
return ERR_CAST(tmp);
|
|
|
|
res = kern_path_create(dfd, tmp, path, is_dir);
|
|
|
|
putname(tmp);
|
|
|
|
return res;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(user_path_create);
|
|
|
|
|
2011-07-26 12:52:52 +07:00
|
|
|
int vfs_mknod(struct inode *dir, struct dentry *dentry, umode_t mode, dev_t dev)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2008-07-30 20:08:48 +07:00
|
|
|
int error = may_create(dir, dentry);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
|
2011-11-14 03:16:43 +07:00
|
|
|
if ((S_ISCHR(mode) || S_ISBLK(mode)) && !capable(CAP_MKNOD))
|
2005-04-17 05:20:36 +07:00
|
|
|
return -EPERM;
|
|
|
|
|
2008-12-04 22:06:33 +07:00
|
|
|
if (!dir->i_op->mknod)
|
2005-04-17 05:20:36 +07:00
|
|
|
return -EPERM;
|
|
|
|
|
2008-04-29 15:00:10 +07:00
|
|
|
error = devcgroup_inode_mknod(mode, dev);
|
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
error = security_inode_mknod(dir, dentry, mode, dev);
|
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
|
|
|
|
error = dir->i_op->mknod(dir, dentry, mode, dev);
|
2005-09-10 03:01:44 +07:00
|
|
|
if (!error)
|
2005-11-03 22:57:06 +07:00
|
|
|
fsnotify_create(dir, dentry);
|
2005-04-17 05:20:36 +07:00
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2011-07-26 15:31:40 +07:00
|
|
|
static int may_mknod(umode_t mode)
|
2008-02-16 05:37:57 +07:00
|
|
|
{
|
|
|
|
switch (mode & S_IFMT) {
|
|
|
|
case S_IFREG:
|
|
|
|
case S_IFCHR:
|
|
|
|
case S_IFBLK:
|
|
|
|
case S_IFIFO:
|
|
|
|
case S_IFSOCK:
|
|
|
|
case 0: /* zero mode translates to S_IFREG */
|
|
|
|
return 0;
|
|
|
|
case S_IFDIR:
|
|
|
|
return -EPERM;
|
|
|
|
default:
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2011-07-26 04:32:17 +07:00
|
|
|
SYSCALL_DEFINE4(mknodat, int, dfd, const char __user *, filename, umode_t, mode,
|
2009-01-14 20:14:31 +07:00
|
|
|
unsigned, dev)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2008-07-21 20:32:51 +07:00
|
|
|
struct dentry *dentry;
|
2011-06-26 22:50:15 +07:00
|
|
|
struct path path;
|
|
|
|
int error;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
if (S_ISDIR(mode))
|
|
|
|
return -EPERM;
|
|
|
|
|
2011-06-26 22:50:15 +07:00
|
|
|
dentry = user_path_create(dfd, filename, &path, 0);
|
|
|
|
if (IS_ERR(dentry))
|
|
|
|
return PTR_ERR(dentry);
|
2008-07-21 20:32:51 +07:00
|
|
|
|
2011-06-26 22:50:15 +07:00
|
|
|
if (!IS_POSIXACL(path.dentry->d_inode))
|
2009-03-30 06:08:22 +07:00
|
|
|
mode &= ~current_umask();
|
2008-02-16 05:37:57 +07:00
|
|
|
error = may_mknod(mode);
|
|
|
|
if (error)
|
|
|
|
goto out_dput;
|
2011-06-26 22:50:15 +07:00
|
|
|
error = mnt_want_write(path.mnt);
|
2008-02-16 05:37:57 +07:00
|
|
|
if (error)
|
|
|
|
goto out_dput;
|
2011-06-26 22:50:15 +07:00
|
|
|
error = security_path_mknod(&path, dentry, mode, dev);
|
2008-12-17 11:24:15 +07:00
|
|
|
if (error)
|
|
|
|
goto out_drop_write;
|
2008-02-16 05:37:57 +07:00
|
|
|
switch (mode & S_IFMT) {
|
2005-04-17 05:20:36 +07:00
|
|
|
case 0: case S_IFREG:
|
2011-06-26 22:50:15 +07:00
|
|
|
error = vfs_create(path.dentry->d_inode,dentry,mode,NULL);
|
2005-04-17 05:20:36 +07:00
|
|
|
break;
|
|
|
|
case S_IFCHR: case S_IFBLK:
|
2011-06-26 22:50:15 +07:00
|
|
|
error = vfs_mknod(path.dentry->d_inode,dentry,mode,
|
2005-04-17 05:20:36 +07:00
|
|
|
new_decode_dev(dev));
|
|
|
|
break;
|
|
|
|
case S_IFIFO: case S_IFSOCK:
|
2011-06-26 22:50:15 +07:00
|
|
|
error = vfs_mknod(path.dentry->d_inode,dentry,mode,0);
|
2005-04-17 05:20:36 +07:00
|
|
|
break;
|
|
|
|
}
|
2008-12-17 11:24:15 +07:00
|
|
|
out_drop_write:
|
2011-06-26 22:50:15 +07:00
|
|
|
mnt_drop_write(path.mnt);
|
2008-02-16 05:37:57 +07:00
|
|
|
out_dput:
|
|
|
|
dput(dentry);
|
2011-06-26 22:50:15 +07:00
|
|
|
mutex_unlock(&path.dentry->d_inode->i_mutex);
|
|
|
|
path_put(&path);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2011-07-26 04:32:17 +07:00
|
|
|
SYSCALL_DEFINE3(mknod, const char __user *, filename, umode_t, mode, unsigned, dev)
|
2006-01-19 08:43:53 +07:00
|
|
|
{
|
|
|
|
return sys_mknodat(AT_FDCWD, filename, mode, dev);
|
|
|
|
}
|
|
|
|
|
2011-07-26 12:41:39 +07:00
|
|
|
int vfs_mkdir(struct inode *dir, struct dentry *dentry, umode_t mode)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2008-07-30 20:08:48 +07:00
|
|
|
int error = may_create(dir, dentry);
|
2012-02-07 00:45:27 +07:00
|
|
|
unsigned max_links = dir->i_sb->s_max_links;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
|
2008-12-04 22:06:33 +07:00
|
|
|
if (!dir->i_op->mkdir)
|
2005-04-17 05:20:36 +07:00
|
|
|
return -EPERM;
|
|
|
|
|
|
|
|
mode &= (S_IRWXUGO|S_ISVTX);
|
|
|
|
error = security_inode_mkdir(dir, dentry, mode);
|
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
|
2012-02-07 00:45:27 +07:00
|
|
|
if (max_links && dir->i_nlink >= max_links)
|
|
|
|
return -EMLINK;
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
error = dir->i_op->mkdir(dir, dentry, mode);
|
2005-09-10 03:01:44 +07:00
|
|
|
if (!error)
|
2005-11-03 22:57:06 +07:00
|
|
|
fsnotify_mkdir(dir, dentry);
|
2005-04-17 05:20:36 +07:00
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2011-11-22 02:59:34 +07:00
|
|
|
SYSCALL_DEFINE3(mkdirat, int, dfd, const char __user *, pathname, umode_t, mode)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2006-10-01 13:29:01 +07:00
|
|
|
struct dentry *dentry;
|
2011-06-26 22:50:15 +07:00
|
|
|
struct path path;
|
|
|
|
int error;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2011-06-26 22:50:15 +07:00
|
|
|
dentry = user_path_create(dfd, pathname, &path, 1);
|
2006-10-01 13:29:01 +07:00
|
|
|
if (IS_ERR(dentry))
|
2011-06-26 22:50:15 +07:00
|
|
|
return PTR_ERR(dentry);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2011-06-26 22:50:15 +07:00
|
|
|
if (!IS_POSIXACL(path.dentry->d_inode))
|
2009-03-30 06:08:22 +07:00
|
|
|
mode &= ~current_umask();
|
2011-06-26 22:50:15 +07:00
|
|
|
error = mnt_want_write(path.mnt);
|
2008-02-16 05:37:57 +07:00
|
|
|
if (error)
|
|
|
|
goto out_dput;
|
2011-06-26 22:50:15 +07:00
|
|
|
error = security_path_mkdir(&path, dentry, mode);
|
2008-12-17 11:24:15 +07:00
|
|
|
if (error)
|
|
|
|
goto out_drop_write;
|
2011-06-26 22:50:15 +07:00
|
|
|
error = vfs_mkdir(path.dentry->d_inode, dentry, mode);
|
2008-12-17 11:24:15 +07:00
|
|
|
out_drop_write:
|
2011-06-26 22:50:15 +07:00
|
|
|
mnt_drop_write(path.mnt);
|
2008-02-16 05:37:57 +07:00
|
|
|
out_dput:
|
2006-10-01 13:29:01 +07:00
|
|
|
dput(dentry);
|
2011-06-26 22:50:15 +07:00
|
|
|
mutex_unlock(&path.dentry->d_inode->i_mutex);
|
|
|
|
path_put(&path);
|
2005-04-17 05:20:36 +07:00
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2011-11-22 02:59:34 +07:00
|
|
|
SYSCALL_DEFINE2(mkdir, const char __user *, pathname, umode_t, mode)
|
2006-01-19 08:43:53 +07:00
|
|
|
{
|
|
|
|
return sys_mkdirat(AT_FDCWD, pathname, mode);
|
|
|
|
}
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
2011-05-25 03:06:08 +07:00
|
|
|
* The dentry_unhash() helper will try to drop the dentry early: we
|
2012-02-15 23:48:40 +07:00
|
|
|
* should have a usage count of 1 if we're the only user of this
|
2011-05-25 03:06:08 +07:00
|
|
|
* dentry, and if that is true (possibly after pruning the dcache),
|
|
|
|
* then we drop the dentry now.
|
2005-04-17 05:20:36 +07:00
|
|
|
*
|
|
|
|
* A low-level filesystem can, if it choses, legally
|
|
|
|
* do a
|
|
|
|
*
|
|
|
|
* if (!d_unhashed(dentry))
|
|
|
|
* return -EBUSY;
|
|
|
|
*
|
|
|
|
* if it cannot handle the case of removing a directory
|
|
|
|
* that is still in use by something else..
|
|
|
|
*/
|
|
|
|
void dentry_unhash(struct dentry *dentry)
|
|
|
|
{
|
2006-12-07 11:37:07 +07:00
|
|
|
shrink_dcache_parent(dentry);
|
2005-04-17 05:20:36 +07:00
|
|
|
spin_lock(&dentry->d_lock);
|
2011-05-25 03:06:05 +07:00
|
|
|
if (dentry->d_count == 1)
|
2005-04-17 05:20:36 +07:00
|
|
|
__d_drop(dentry);
|
|
|
|
spin_unlock(&dentry->d_lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
int vfs_rmdir(struct inode *dir, struct dentry *dentry)
|
|
|
|
{
|
|
|
|
int error = may_delete(dir, dentry, 1);
|
|
|
|
|
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
|
2008-12-04 22:06:33 +07:00
|
|
|
if (!dir->i_op->rmdir)
|
2005-04-17 05:20:36 +07:00
|
|
|
return -EPERM;
|
|
|
|
|
2011-09-15 00:55:41 +07:00
|
|
|
dget(dentry);
|
2006-01-10 06:59:24 +07:00
|
|
|
mutex_lock(&dentry->d_inode->i_mutex);
|
2011-05-25 03:06:11 +07:00
|
|
|
|
|
|
|
error = -EBUSY;
|
2005-04-17 05:20:36 +07:00
|
|
|
if (d_mountpoint(dentry))
|
2011-05-25 03:06:11 +07:00
|
|
|
goto out;
|
|
|
|
|
|
|
|
error = security_inode_rmdir(dir, dentry);
|
|
|
|
if (error)
|
|
|
|
goto out;
|
|
|
|
|
2011-05-30 11:20:59 +07:00
|
|
|
shrink_dcache_parent(dentry);
|
2011-05-25 03:06:11 +07:00
|
|
|
error = dir->i_op->rmdir(dir, dentry);
|
|
|
|
if (error)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
dentry->d_inode->i_flags |= S_DEAD;
|
|
|
|
dont_mount(dentry);
|
|
|
|
|
|
|
|
out:
|
2006-01-10 06:59:24 +07:00
|
|
|
mutex_unlock(&dentry->d_inode->i_mutex);
|
2011-09-15 00:55:41 +07:00
|
|
|
dput(dentry);
|
2011-05-25 03:06:11 +07:00
|
|
|
if (!error)
|
2005-04-17 05:20:36 +07:00
|
|
|
d_delete(dentry);
|
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2006-01-19 08:43:53 +07:00
|
|
|
static long do_rmdir(int dfd, const char __user *pathname)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
int error = 0;
|
|
|
|
char * name;
|
|
|
|
struct dentry *dentry;
|
|
|
|
struct nameidata nd;
|
|
|
|
|
2008-07-21 20:32:51 +07:00
|
|
|
error = user_path_parent(dfd, pathname, &nd, &name);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (error)
|
2008-07-21 20:32:51 +07:00
|
|
|
return error;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
switch(nd.last_type) {
|
2008-10-16 05:50:29 +07:00
|
|
|
case LAST_DOTDOT:
|
|
|
|
error = -ENOTEMPTY;
|
|
|
|
goto exit1;
|
|
|
|
case LAST_DOT:
|
|
|
|
error = -EINVAL;
|
|
|
|
goto exit1;
|
|
|
|
case LAST_ROOT:
|
|
|
|
error = -EBUSY;
|
|
|
|
goto exit1;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
2008-10-16 05:50:29 +07:00
|
|
|
|
|
|
|
nd.flags &= ~LOOKUP_PARENT;
|
|
|
|
|
2008-02-15 10:34:32 +07:00
|
|
|
mutex_lock_nested(&nd.path.dentry->d_inode->i_mutex, I_MUTEX_PARENT);
|
2005-11-09 12:35:06 +07:00
|
|
|
dentry = lookup_hash(&nd);
|
2005-04-17 05:20:36 +07:00
|
|
|
error = PTR_ERR(dentry);
|
2006-10-01 13:29:01 +07:00
|
|
|
if (IS_ERR(dentry))
|
|
|
|
goto exit2;
|
2011-06-07 06:19:40 +07:00
|
|
|
if (!dentry->d_inode) {
|
|
|
|
error = -ENOENT;
|
|
|
|
goto exit3;
|
|
|
|
}
|
2008-02-16 05:37:34 +07:00
|
|
|
error = mnt_want_write(nd.path.mnt);
|
|
|
|
if (error)
|
|
|
|
goto exit3;
|
2008-12-17 11:24:15 +07:00
|
|
|
error = security_path_rmdir(&nd.path, dentry);
|
|
|
|
if (error)
|
|
|
|
goto exit4;
|
2008-02-15 10:34:32 +07:00
|
|
|
error = vfs_rmdir(nd.path.dentry->d_inode, dentry);
|
2008-12-17 11:24:15 +07:00
|
|
|
exit4:
|
2008-02-16 05:37:34 +07:00
|
|
|
mnt_drop_write(nd.path.mnt);
|
|
|
|
exit3:
|
2006-10-01 13:29:01 +07:00
|
|
|
dput(dentry);
|
|
|
|
exit2:
|
2008-02-15 10:34:32 +07:00
|
|
|
mutex_unlock(&nd.path.dentry->d_inode->i_mutex);
|
2005-04-17 05:20:36 +07:00
|
|
|
exit1:
|
2008-02-15 10:34:35 +07:00
|
|
|
path_put(&nd.path);
|
2005-04-17 05:20:36 +07:00
|
|
|
putname(name);
|
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2009-01-14 20:14:22 +07:00
|
|
|
SYSCALL_DEFINE1(rmdir, const char __user *, pathname)
|
2006-01-19 08:43:53 +07:00
|
|
|
{
|
|
|
|
return do_rmdir(AT_FDCWD, pathname);
|
|
|
|
}
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
int vfs_unlink(struct inode *dir, struct dentry *dentry)
|
|
|
|
{
|
|
|
|
int error = may_delete(dir, dentry, 0);
|
|
|
|
|
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
|
2008-12-04 22:06:33 +07:00
|
|
|
if (!dir->i_op->unlink)
|
2005-04-17 05:20:36 +07:00
|
|
|
return -EPERM;
|
|
|
|
|
2006-01-10 06:59:24 +07:00
|
|
|
mutex_lock(&dentry->d_inode->i_mutex);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (d_mountpoint(dentry))
|
|
|
|
error = -EBUSY;
|
|
|
|
else {
|
|
|
|
error = security_inode_unlink(dir, dentry);
|
2010-03-04 02:12:08 +07:00
|
|
|
if (!error) {
|
2005-04-17 05:20:36 +07:00
|
|
|
error = dir->i_op->unlink(dir, dentry);
|
2010-03-04 02:12:08 +07:00
|
|
|
if (!error)
|
2010-05-01 04:17:09 +07:00
|
|
|
dont_mount(dentry);
|
2010-03-04 02:12:08 +07:00
|
|
|
}
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
2006-01-10 06:59:24 +07:00
|
|
|
mutex_unlock(&dentry->d_inode->i_mutex);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
/* We don't d_delete() NFS sillyrenamed files--they still exist. */
|
|
|
|
if (!error && !(dentry->d_flags & DCACHE_NFSFS_RENAMED)) {
|
2008-02-06 16:37:13 +07:00
|
|
|
fsnotify_link_count(dentry->d_inode);
|
2005-08-05 03:07:08 +07:00
|
|
|
d_delete(dentry);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
[PATCH] inotify
inotify is intended to correct the deficiencies of dnotify, particularly
its inability to scale and its terrible user interface:
* dnotify requires the opening of one fd per each directory
that you intend to watch. This quickly results in too many
open files and pins removable media, preventing unmount.
* dnotify is directory-based. You only learn about changes to
directories. Sure, a change to a file in a directory affects
the directory, but you are then forced to keep a cache of
stat structures.
* dnotify's interface to user-space is awful. Signals?
inotify provides a more usable, simple, powerful solution to file change
notification:
* inotify's interface is a system call that returns a fd, not SIGIO.
You get a single fd, which is select()-able.
* inotify has an event that says "the filesystem that the item
you were watching is on was unmounted."
* inotify can watch directories or files.
Inotify is currently used by Beagle (a desktop search infrastructure),
Gamin (a FAM replacement), and other projects.
See Documentation/filesystems/inotify.txt.
Signed-off-by: Robert Love <rml@novell.com>
Cc: John McCutchan <ttb@tentacle.dhs.org>
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-07-13 04:06:03 +07:00
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Make sure that the actual truncation of the file will occur outside its
|
2006-01-10 06:59:24 +07:00
|
|
|
* directory's i_mutex. Truncate can take a long time if there is a lot of
|
2005-04-17 05:20:36 +07:00
|
|
|
* writeout happening, and we don't want to prevent access to the directory
|
|
|
|
* while waiting on the I/O.
|
|
|
|
*/
|
2006-01-19 08:43:53 +07:00
|
|
|
static long do_unlinkat(int dfd, const char __user *pathname)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2008-07-21 20:32:51 +07:00
|
|
|
int error;
|
|
|
|
char *name;
|
2005-04-17 05:20:36 +07:00
|
|
|
struct dentry *dentry;
|
|
|
|
struct nameidata nd;
|
|
|
|
struct inode *inode = NULL;
|
|
|
|
|
2008-07-21 20:32:51 +07:00
|
|
|
error = user_path_parent(dfd, pathname, &nd, &name);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (error)
|
2008-07-21 20:32:51 +07:00
|
|
|
return error;
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
error = -EISDIR;
|
|
|
|
if (nd.last_type != LAST_NORM)
|
|
|
|
goto exit1;
|
2008-10-16 05:50:29 +07:00
|
|
|
|
|
|
|
nd.flags &= ~LOOKUP_PARENT;
|
|
|
|
|
2008-02-15 10:34:32 +07:00
|
|
|
mutex_lock_nested(&nd.path.dentry->d_inode->i_mutex, I_MUTEX_PARENT);
|
2005-11-09 12:35:06 +07:00
|
|
|
dentry = lookup_hash(&nd);
|
2005-04-17 05:20:36 +07:00
|
|
|
error = PTR_ERR(dentry);
|
|
|
|
if (!IS_ERR(dentry)) {
|
|
|
|
/* Why not before? Because we want correct error value */
|
2011-06-16 04:06:14 +07:00
|
|
|
if (nd.last.name[nd.last.len])
|
|
|
|
goto slashes;
|
2005-04-17 05:20:36 +07:00
|
|
|
inode = dentry->d_inode;
|
2011-06-16 04:06:14 +07:00
|
|
|
if (!inode)
|
2011-06-07 06:19:40 +07:00
|
|
|
goto slashes;
|
|
|
|
ihold(inode);
|
2008-02-16 05:37:34 +07:00
|
|
|
error = mnt_want_write(nd.path.mnt);
|
|
|
|
if (error)
|
|
|
|
goto exit2;
|
2008-12-17 11:24:15 +07:00
|
|
|
error = security_path_unlink(&nd.path, dentry);
|
|
|
|
if (error)
|
|
|
|
goto exit3;
|
2008-02-15 10:34:32 +07:00
|
|
|
error = vfs_unlink(nd.path.dentry->d_inode, dentry);
|
2008-12-17 11:24:15 +07:00
|
|
|
exit3:
|
2008-02-16 05:37:34 +07:00
|
|
|
mnt_drop_write(nd.path.mnt);
|
2005-04-17 05:20:36 +07:00
|
|
|
exit2:
|
|
|
|
dput(dentry);
|
|
|
|
}
|
2008-02-15 10:34:32 +07:00
|
|
|
mutex_unlock(&nd.path.dentry->d_inode->i_mutex);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (inode)
|
|
|
|
iput(inode); /* truncate the inode here */
|
|
|
|
exit1:
|
2008-02-15 10:34:35 +07:00
|
|
|
path_put(&nd.path);
|
2005-04-17 05:20:36 +07:00
|
|
|
putname(name);
|
|
|
|
return error;
|
|
|
|
|
|
|
|
slashes:
|
|
|
|
error = !dentry->d_inode ? -ENOENT :
|
|
|
|
S_ISDIR(dentry->d_inode->i_mode) ? -EISDIR : -ENOTDIR;
|
|
|
|
goto exit2;
|
|
|
|
}
|
|
|
|
|
2009-01-14 20:14:31 +07:00
|
|
|
SYSCALL_DEFINE3(unlinkat, int, dfd, const char __user *, pathname, int, flag)
|
2006-01-19 08:43:53 +07:00
|
|
|
{
|
|
|
|
if ((flag & ~AT_REMOVEDIR) != 0)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
if (flag & AT_REMOVEDIR)
|
|
|
|
return do_rmdir(dfd, pathname);
|
|
|
|
|
|
|
|
return do_unlinkat(dfd, pathname);
|
|
|
|
}
|
|
|
|
|
2009-01-14 20:14:16 +07:00
|
|
|
SYSCALL_DEFINE1(unlink, const char __user *, pathname)
|
2006-01-19 08:43:53 +07:00
|
|
|
{
|
|
|
|
return do_unlinkat(AT_FDCWD, pathname);
|
|
|
|
}
|
|
|
|
|
2008-06-24 21:50:16 +07:00
|
|
|
int vfs_symlink(struct inode *dir, struct dentry *dentry, const char *oldname)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2008-07-30 20:08:48 +07:00
|
|
|
int error = may_create(dir, dentry);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
|
2008-12-04 22:06:33 +07:00
|
|
|
if (!dir->i_op->symlink)
|
2005-04-17 05:20:36 +07:00
|
|
|
return -EPERM;
|
|
|
|
|
|
|
|
error = security_inode_symlink(dir, dentry, oldname);
|
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
|
|
|
|
error = dir->i_op->symlink(dir, dentry, oldname);
|
2005-09-10 03:01:44 +07:00
|
|
|
if (!error)
|
2005-11-03 22:57:06 +07:00
|
|
|
fsnotify_create(dir, dentry);
|
2005-04-17 05:20:36 +07:00
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2009-01-14 20:14:31 +07:00
|
|
|
SYSCALL_DEFINE3(symlinkat, const char __user *, oldname,
|
|
|
|
int, newdfd, const char __user *, newname)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2008-07-21 20:32:51 +07:00
|
|
|
int error;
|
|
|
|
char *from;
|
2006-10-01 13:29:01 +07:00
|
|
|
struct dentry *dentry;
|
2011-06-26 22:50:15 +07:00
|
|
|
struct path path;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
from = getname(oldname);
|
2008-07-21 20:32:51 +07:00
|
|
|
if (IS_ERR(from))
|
2005-04-17 05:20:36 +07:00
|
|
|
return PTR_ERR(from);
|
|
|
|
|
2011-06-26 22:50:15 +07:00
|
|
|
dentry = user_path_create(newdfd, newname, &path, 0);
|
2006-10-01 13:29:01 +07:00
|
|
|
error = PTR_ERR(dentry);
|
|
|
|
if (IS_ERR(dentry))
|
2011-06-26 22:50:15 +07:00
|
|
|
goto out_putname;
|
2006-10-01 13:29:01 +07:00
|
|
|
|
2011-06-26 22:50:15 +07:00
|
|
|
error = mnt_want_write(path.mnt);
|
2008-02-16 05:37:45 +07:00
|
|
|
if (error)
|
|
|
|
goto out_dput;
|
2011-06-26 22:50:15 +07:00
|
|
|
error = security_path_symlink(&path, dentry, from);
|
2008-12-17 11:24:15 +07:00
|
|
|
if (error)
|
|
|
|
goto out_drop_write;
|
2011-06-26 22:50:15 +07:00
|
|
|
error = vfs_symlink(path.dentry->d_inode, dentry, from);
|
2008-12-17 11:24:15 +07:00
|
|
|
out_drop_write:
|
2011-06-26 22:50:15 +07:00
|
|
|
mnt_drop_write(path.mnt);
|
2008-02-16 05:37:45 +07:00
|
|
|
out_dput:
|
2006-10-01 13:29:01 +07:00
|
|
|
dput(dentry);
|
2011-06-26 22:50:15 +07:00
|
|
|
mutex_unlock(&path.dentry->d_inode->i_mutex);
|
|
|
|
path_put(&path);
|
2006-10-01 13:29:01 +07:00
|
|
|
out_putname:
|
2005-04-17 05:20:36 +07:00
|
|
|
putname(from);
|
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2009-01-14 20:14:16 +07:00
|
|
|
SYSCALL_DEFINE2(symlink, const char __user *, oldname, const char __user *, newname)
|
2006-01-19 08:43:53 +07:00
|
|
|
{
|
|
|
|
return sys_symlinkat(oldname, AT_FDCWD, newname);
|
|
|
|
}
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
int vfs_link(struct dentry *old_dentry, struct inode *dir, struct dentry *new_dentry)
|
|
|
|
{
|
|
|
|
struct inode *inode = old_dentry->d_inode;
|
2012-02-07 00:45:27 +07:00
|
|
|
unsigned max_links = dir->i_sb->s_max_links;
|
2005-04-17 05:20:36 +07:00
|
|
|
int error;
|
|
|
|
|
|
|
|
if (!inode)
|
|
|
|
return -ENOENT;
|
|
|
|
|
2008-07-30 20:08:48 +07:00
|
|
|
error = may_create(dir, new_dentry);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
|
|
|
|
if (dir->i_sb != inode->i_sb)
|
|
|
|
return -EXDEV;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* A link to an append-only or immutable file cannot be created.
|
|
|
|
*/
|
|
|
|
if (IS_APPEND(inode) || IS_IMMUTABLE(inode))
|
|
|
|
return -EPERM;
|
2008-12-04 22:06:33 +07:00
|
|
|
if (!dir->i_op->link)
|
2005-04-17 05:20:36 +07:00
|
|
|
return -EPERM;
|
2008-06-24 21:50:15 +07:00
|
|
|
if (S_ISDIR(inode->i_mode))
|
2005-04-17 05:20:36 +07:00
|
|
|
return -EPERM;
|
|
|
|
|
|
|
|
error = security_inode_link(old_dentry, dir, new_dentry);
|
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
|
2008-06-24 21:50:15 +07:00
|
|
|
mutex_lock(&inode->i_mutex);
|
2011-01-29 20:13:27 +07:00
|
|
|
/* Make sure we don't allow creating hardlink to an unlinked file */
|
|
|
|
if (inode->i_nlink == 0)
|
|
|
|
error = -ENOENT;
|
2012-02-07 00:45:27 +07:00
|
|
|
else if (max_links && inode->i_nlink >= max_links)
|
|
|
|
error = -EMLINK;
|
2011-01-29 20:13:27 +07:00
|
|
|
else
|
|
|
|
error = dir->i_op->link(old_dentry, dir, new_dentry);
|
2008-06-24 21:50:15 +07:00
|
|
|
mutex_unlock(&inode->i_mutex);
|
2005-09-10 03:01:45 +07:00
|
|
|
if (!error)
|
2008-06-24 21:50:15 +07:00
|
|
|
fsnotify_link(dir, inode, new_dentry);
|
2005-04-17 05:20:36 +07:00
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Hardlinks are often used in delicate situations. We avoid
|
|
|
|
* security-related surprises by not following symlinks on the
|
|
|
|
* newname. --KAB
|
|
|
|
*
|
|
|
|
* We don't follow them on the oldname either to be compatible
|
|
|
|
* with linux 2.0, and to avoid hard-linking to directories
|
|
|
|
* and other special files. --ADM
|
|
|
|
*/
|
2009-01-14 20:14:31 +07:00
|
|
|
SYSCALL_DEFINE5(linkat, int, olddfd, const char __user *, oldname,
|
|
|
|
int, newdfd, const char __user *, newname, int, flags)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
struct dentry *new_dentry;
|
2011-06-26 22:50:15 +07:00
|
|
|
struct path old_path, new_path;
|
2011-01-29 20:13:42 +07:00
|
|
|
int how = 0;
|
2005-04-17 05:20:36 +07:00
|
|
|
int error;
|
|
|
|
|
2011-01-29 20:13:42 +07:00
|
|
|
if ((flags & ~(AT_SYMLINK_FOLLOW | AT_EMPTY_PATH)) != 0)
|
2006-02-25 04:04:21 +07:00
|
|
|
return -EINVAL;
|
2011-01-29 20:13:42 +07:00
|
|
|
/*
|
|
|
|
* To use null names we require CAP_DAC_READ_SEARCH
|
|
|
|
* This ensures that not everyone will be able to create
|
|
|
|
* handlink using the passed filedescriptor.
|
|
|
|
*/
|
|
|
|
if (flags & AT_EMPTY_PATH) {
|
|
|
|
if (!capable(CAP_DAC_READ_SEARCH))
|
|
|
|
return -ENOENT;
|
|
|
|
how = LOOKUP_EMPTY;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (flags & AT_SYMLINK_FOLLOW)
|
|
|
|
how |= LOOKUP_FOLLOW;
|
2006-02-25 04:04:21 +07:00
|
|
|
|
2011-01-29 20:13:42 +07:00
|
|
|
error = user_path_at(olddfd, oldname, how, &old_path);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (error)
|
2008-07-21 20:32:51 +07:00
|
|
|
return error;
|
|
|
|
|
2011-06-26 22:50:15 +07:00
|
|
|
new_dentry = user_path_create(newdfd, newname, &new_path, 0);
|
2005-04-17 05:20:36 +07:00
|
|
|
error = PTR_ERR(new_dentry);
|
2006-10-01 13:29:01 +07:00
|
|
|
if (IS_ERR(new_dentry))
|
2011-06-26 22:50:15 +07:00
|
|
|
goto out;
|
|
|
|
|
|
|
|
error = -EXDEV;
|
|
|
|
if (old_path.mnt != new_path.mnt)
|
|
|
|
goto out_dput;
|
|
|
|
error = mnt_want_write(new_path.mnt);
|
2008-02-16 05:37:45 +07:00
|
|
|
if (error)
|
|
|
|
goto out_dput;
|
2011-06-26 22:50:15 +07:00
|
|
|
error = security_path_link(old_path.dentry, &new_path, new_dentry);
|
2008-12-17 11:24:15 +07:00
|
|
|
if (error)
|
|
|
|
goto out_drop_write;
|
2011-06-26 22:50:15 +07:00
|
|
|
error = vfs_link(old_path.dentry, new_path.dentry->d_inode, new_dentry);
|
2008-12-17 11:24:15 +07:00
|
|
|
out_drop_write:
|
2011-06-26 22:50:15 +07:00
|
|
|
mnt_drop_write(new_path.mnt);
|
2008-02-16 05:37:45 +07:00
|
|
|
out_dput:
|
2006-10-01 13:29:01 +07:00
|
|
|
dput(new_dentry);
|
2011-06-26 22:50:15 +07:00
|
|
|
mutex_unlock(&new_path.dentry->d_inode->i_mutex);
|
|
|
|
path_put(&new_path);
|
2005-04-17 05:20:36 +07:00
|
|
|
out:
|
2008-07-22 20:59:21 +07:00
|
|
|
path_put(&old_path);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2009-01-14 20:14:16 +07:00
|
|
|
SYSCALL_DEFINE2(link, const char __user *, oldname, const char __user *, newname)
|
2006-01-19 08:43:53 +07:00
|
|
|
{
|
2006-02-25 04:04:21 +07:00
|
|
|
return sys_linkat(AT_FDCWD, oldname, AT_FDCWD, newname, 0);
|
2006-01-19 08:43:53 +07:00
|
|
|
}
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* The worst of all namespace operations - renaming directory. "Perverted"
|
|
|
|
* doesn't even start to describe it. Somebody in UCB had a heck of a trip...
|
|
|
|
* Problems:
|
|
|
|
* a) we can get into loop creation. Check is done in is_subdir().
|
|
|
|
* b) race potential - two innocent renames can create a loop together.
|
|
|
|
* That's where 4.4 screws up. Current fix: serialization on
|
2006-03-23 18:00:33 +07:00
|
|
|
* sb->s_vfs_rename_mutex. We might be more accurate, but that's another
|
2005-04-17 05:20:36 +07:00
|
|
|
* story.
|
|
|
|
* c) we have to lock _three_ objects - parents and victim (if it exists).
|
2006-01-10 06:59:24 +07:00
|
|
|
* And that - after we got ->i_mutex on parents (until then we don't know
|
2005-04-17 05:20:36 +07:00
|
|
|
* whether the target exists). Solution: try to be smart with locking
|
|
|
|
* order for inodes. We rely on the fact that tree topology may change
|
2006-03-23 18:00:33 +07:00
|
|
|
* only under ->s_vfs_rename_mutex _and_ that parent of the object we
|
2005-04-17 05:20:36 +07:00
|
|
|
* move will be locked. Thus we can rank directories by the tree
|
|
|
|
* (ancestors first) and rank all non-directories after them.
|
|
|
|
* That works since everybody except rename does "lock parent, lookup,
|
2006-03-23 18:00:33 +07:00
|
|
|
* lock child" and rename is under ->s_vfs_rename_mutex.
|
2005-04-17 05:20:36 +07:00
|
|
|
* HOWEVER, it relies on the assumption that any object with ->lookup()
|
|
|
|
* has no more than 1 dentry. If "hybrid" objects will ever appear,
|
|
|
|
* we'd better make sure that there's no link(2) for them.
|
2011-05-25 03:06:07 +07:00
|
|
|
* d) conversion from fhandle to dentry may come in the wrong moment - when
|
2006-01-10 06:59:24 +07:00
|
|
|
* we are removing the target. Solution: we will have to grab ->i_mutex
|
2005-04-17 05:20:36 +07:00
|
|
|
* in the fhandle_to_dentry code. [FIXME - current nfsfh.c relies on
|
2009-12-12 04:35:39 +07:00
|
|
|
* ->i_mutex on parents, which works but leads to some truly excessive
|
2005-04-17 05:20:36 +07:00
|
|
|
* locking].
|
|
|
|
*/
|
2005-05-06 06:16:09 +07:00
|
|
|
static int vfs_rename_dir(struct inode *old_dir, struct dentry *old_dentry,
|
|
|
|
struct inode *new_dir, struct dentry *new_dentry)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
int error = 0;
|
2011-05-25 03:06:12 +07:00
|
|
|
struct inode *target = new_dentry->d_inode;
|
2012-02-07 00:45:27 +07:00
|
|
|
unsigned max_links = new_dir->i_sb->s_max_links;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If we are going to change the parent - check write permissions,
|
|
|
|
* we'll need to flip '..'.
|
|
|
|
*/
|
|
|
|
if (new_dir != old_dir) {
|
2008-07-22 11:07:17 +07:00
|
|
|
error = inode_permission(old_dentry->d_inode, MAY_WRITE);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
|
|
|
error = security_inode_rename(old_dir, old_dentry, new_dir, new_dentry);
|
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
|
2011-09-15 00:55:41 +07:00
|
|
|
dget(new_dentry);
|
2010-05-01 04:17:09 +07:00
|
|
|
if (target)
|
2006-01-10 06:59:24 +07:00
|
|
|
mutex_lock(&target->i_mutex);
|
2011-05-25 03:06:12 +07:00
|
|
|
|
|
|
|
error = -EBUSY;
|
|
|
|
if (d_mountpoint(old_dentry) || d_mountpoint(new_dentry))
|
|
|
|
goto out;
|
|
|
|
|
2012-02-07 00:45:27 +07:00
|
|
|
error = -EMLINK;
|
|
|
|
if (max_links && !target && new_dir != old_dir &&
|
|
|
|
new_dir->i_nlink >= max_links)
|
|
|
|
goto out;
|
|
|
|
|
2011-05-30 11:20:59 +07:00
|
|
|
if (target)
|
|
|
|
shrink_dcache_parent(new_dentry);
|
2011-05-25 03:06:12 +07:00
|
|
|
error = old_dir->i_op->rename(old_dir, old_dentry, new_dir, new_dentry);
|
|
|
|
if (error)
|
|
|
|
goto out;
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
if (target) {
|
2011-05-25 03:06:12 +07:00
|
|
|
target->i_flags |= S_DEAD;
|
|
|
|
dont_mount(new_dentry);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
2011-05-25 03:06:12 +07:00
|
|
|
out:
|
|
|
|
if (target)
|
|
|
|
mutex_unlock(&target->i_mutex);
|
2011-09-15 00:55:41 +07:00
|
|
|
dput(new_dentry);
|
2005-09-10 03:01:45 +07:00
|
|
|
if (!error)
|
2006-09-09 04:22:21 +07:00
|
|
|
if (!(old_dir->i_sb->s_type->fs_flags & FS_RENAME_DOES_D_MOVE))
|
|
|
|
d_move(old_dentry,new_dentry);
|
2005-04-17 05:20:36 +07:00
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2005-05-06 06:16:09 +07:00
|
|
|
static int vfs_rename_other(struct inode *old_dir, struct dentry *old_dentry,
|
|
|
|
struct inode *new_dir, struct dentry *new_dentry)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2011-05-25 03:06:13 +07:00
|
|
|
struct inode *target = new_dentry->d_inode;
|
2005-04-17 05:20:36 +07:00
|
|
|
int error;
|
|
|
|
|
|
|
|
error = security_inode_rename(old_dir, old_dentry, new_dir, new_dentry);
|
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
|
|
|
|
dget(new_dentry);
|
|
|
|
if (target)
|
2006-01-10 06:59:24 +07:00
|
|
|
mutex_lock(&target->i_mutex);
|
2011-05-25 03:06:13 +07:00
|
|
|
|
|
|
|
error = -EBUSY;
|
2005-04-17 05:20:36 +07:00
|
|
|
if (d_mountpoint(old_dentry)||d_mountpoint(new_dentry))
|
2011-05-25 03:06:13 +07:00
|
|
|
goto out;
|
|
|
|
|
|
|
|
error = old_dir->i_op->rename(old_dir, old_dentry, new_dir, new_dentry);
|
|
|
|
if (error)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
if (target)
|
|
|
|
dont_mount(new_dentry);
|
|
|
|
if (!(old_dir->i_sb->s_type->fs_flags & FS_RENAME_DOES_D_MOVE))
|
|
|
|
d_move(old_dentry, new_dentry);
|
|
|
|
out:
|
2005-04-17 05:20:36 +07:00
|
|
|
if (target)
|
2006-01-10 06:59:24 +07:00
|
|
|
mutex_unlock(&target->i_mutex);
|
2005-04-17 05:20:36 +07:00
|
|
|
dput(new_dentry);
|
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
|
|
|
int vfs_rename(struct inode *old_dir, struct dentry *old_dentry,
|
|
|
|
struct inode *new_dir, struct dentry *new_dentry)
|
|
|
|
{
|
|
|
|
int error;
|
|
|
|
int is_dir = S_ISDIR(old_dentry->d_inode->i_mode);
|
2010-02-09 00:53:52 +07:00
|
|
|
const unsigned char *old_name;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
if (old_dentry->d_inode == new_dentry->d_inode)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
error = may_delete(old_dir, old_dentry, is_dir);
|
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
|
|
|
|
if (!new_dentry->d_inode)
|
2008-07-30 20:08:48 +07:00
|
|
|
error = may_create(new_dir, new_dentry);
|
2005-04-17 05:20:36 +07:00
|
|
|
else
|
|
|
|
error = may_delete(new_dir, new_dentry, is_dir);
|
|
|
|
if (error)
|
|
|
|
return error;
|
|
|
|
|
2008-12-04 22:06:33 +07:00
|
|
|
if (!old_dir->i_op->rename)
|
2005-04-17 05:20:36 +07:00
|
|
|
return -EPERM;
|
|
|
|
|
[PATCH] inotify
inotify is intended to correct the deficiencies of dnotify, particularly
its inability to scale and its terrible user interface:
* dnotify requires the opening of one fd per each directory
that you intend to watch. This quickly results in too many
open files and pins removable media, preventing unmount.
* dnotify is directory-based. You only learn about changes to
directories. Sure, a change to a file in a directory affects
the directory, but you are then forced to keep a cache of
stat structures.
* dnotify's interface to user-space is awful. Signals?
inotify provides a more usable, simple, powerful solution to file change
notification:
* inotify's interface is a system call that returns a fd, not SIGIO.
You get a single fd, which is select()-able.
* inotify has an event that says "the filesystem that the item
you were watching is on was unmounted."
* inotify can watch directories or files.
Inotify is currently used by Beagle (a desktop search infrastructure),
Gamin (a FAM replacement), and other projects.
See Documentation/filesystems/inotify.txt.
Signed-off-by: Robert Love <rml@novell.com>
Cc: John McCutchan <ttb@tentacle.dhs.org>
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-07-13 04:06:03 +07:00
|
|
|
old_name = fsnotify_oldname_init(old_dentry->d_name.name);
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
if (is_dir)
|
|
|
|
error = vfs_rename_dir(old_dir,old_dentry,new_dir,new_dentry);
|
|
|
|
else
|
|
|
|
error = vfs_rename_other(old_dir,old_dentry,new_dir,new_dentry);
|
2009-12-25 16:57:57 +07:00
|
|
|
if (!error)
|
|
|
|
fsnotify_move(old_dir, new_dir, old_name, is_dir,
|
2007-06-07 23:19:32 +07:00
|
|
|
new_dentry->d_inode, old_dentry);
|
[PATCH] inotify
inotify is intended to correct the deficiencies of dnotify, particularly
its inability to scale and its terrible user interface:
* dnotify requires the opening of one fd per each directory
that you intend to watch. This quickly results in too many
open files and pins removable media, preventing unmount.
* dnotify is directory-based. You only learn about changes to
directories. Sure, a change to a file in a directory affects
the directory, but you are then forced to keep a cache of
stat structures.
* dnotify's interface to user-space is awful. Signals?
inotify provides a more usable, simple, powerful solution to file change
notification:
* inotify's interface is a system call that returns a fd, not SIGIO.
You get a single fd, which is select()-able.
* inotify has an event that says "the filesystem that the item
you were watching is on was unmounted."
* inotify can watch directories or files.
Inotify is currently used by Beagle (a desktop search infrastructure),
Gamin (a FAM replacement), and other projects.
See Documentation/filesystems/inotify.txt.
Signed-off-by: Robert Love <rml@novell.com>
Cc: John McCutchan <ttb@tentacle.dhs.org>
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-07-13 04:06:03 +07:00
|
|
|
fsnotify_oldname_free(old_name);
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2009-01-14 20:14:31 +07:00
|
|
|
SYSCALL_DEFINE4(renameat, int, olddfd, const char __user *, oldname,
|
|
|
|
int, newdfd, const char __user *, newname)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2008-07-21 20:32:51 +07:00
|
|
|
struct dentry *old_dir, *new_dir;
|
|
|
|
struct dentry *old_dentry, *new_dentry;
|
|
|
|
struct dentry *trap;
|
2005-04-17 05:20:36 +07:00
|
|
|
struct nameidata oldnd, newnd;
|
2008-07-21 20:32:51 +07:00
|
|
|
char *from;
|
|
|
|
char *to;
|
|
|
|
int error;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2008-07-21 20:32:51 +07:00
|
|
|
error = user_path_parent(olddfd, oldname, &oldnd, &from);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (error)
|
|
|
|
goto exit;
|
|
|
|
|
2008-07-21 20:32:51 +07:00
|
|
|
error = user_path_parent(newdfd, newname, &newnd, &to);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (error)
|
|
|
|
goto exit1;
|
|
|
|
|
|
|
|
error = -EXDEV;
|
2008-02-15 10:34:32 +07:00
|
|
|
if (oldnd.path.mnt != newnd.path.mnt)
|
2005-04-17 05:20:36 +07:00
|
|
|
goto exit2;
|
|
|
|
|
2008-02-15 10:34:32 +07:00
|
|
|
old_dir = oldnd.path.dentry;
|
2005-04-17 05:20:36 +07:00
|
|
|
error = -EBUSY;
|
|
|
|
if (oldnd.last_type != LAST_NORM)
|
|
|
|
goto exit2;
|
|
|
|
|
2008-02-15 10:34:32 +07:00
|
|
|
new_dir = newnd.path.dentry;
|
2005-04-17 05:20:36 +07:00
|
|
|
if (newnd.last_type != LAST_NORM)
|
|
|
|
goto exit2;
|
|
|
|
|
2008-10-16 05:50:29 +07:00
|
|
|
oldnd.flags &= ~LOOKUP_PARENT;
|
|
|
|
newnd.flags &= ~LOOKUP_PARENT;
|
2008-10-16 05:50:29 +07:00
|
|
|
newnd.flags |= LOOKUP_RENAME_TARGET;
|
2008-10-16 05:50:29 +07:00
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
trap = lock_rename(new_dir, old_dir);
|
|
|
|
|
2005-11-09 12:35:06 +07:00
|
|
|
old_dentry = lookup_hash(&oldnd);
|
2005-04-17 05:20:36 +07:00
|
|
|
error = PTR_ERR(old_dentry);
|
|
|
|
if (IS_ERR(old_dentry))
|
|
|
|
goto exit3;
|
|
|
|
/* source must exist */
|
|
|
|
error = -ENOENT;
|
|
|
|
if (!old_dentry->d_inode)
|
|
|
|
goto exit4;
|
|
|
|
/* unless the source is a directory trailing slashes give -ENOTDIR */
|
|
|
|
if (!S_ISDIR(old_dentry->d_inode->i_mode)) {
|
|
|
|
error = -ENOTDIR;
|
|
|
|
if (oldnd.last.name[oldnd.last.len])
|
|
|
|
goto exit4;
|
|
|
|
if (newnd.last.name[newnd.last.len])
|
|
|
|
goto exit4;
|
|
|
|
}
|
|
|
|
/* source should not be ancestor of target */
|
|
|
|
error = -EINVAL;
|
|
|
|
if (old_dentry == trap)
|
|
|
|
goto exit4;
|
2005-11-09 12:35:06 +07:00
|
|
|
new_dentry = lookup_hash(&newnd);
|
2005-04-17 05:20:36 +07:00
|
|
|
error = PTR_ERR(new_dentry);
|
|
|
|
if (IS_ERR(new_dentry))
|
|
|
|
goto exit4;
|
|
|
|
/* target should not be an ancestor of source */
|
|
|
|
error = -ENOTEMPTY;
|
|
|
|
if (new_dentry == trap)
|
|
|
|
goto exit5;
|
|
|
|
|
2008-02-16 05:37:49 +07:00
|
|
|
error = mnt_want_write(oldnd.path.mnt);
|
|
|
|
if (error)
|
|
|
|
goto exit5;
|
2008-12-17 11:24:15 +07:00
|
|
|
error = security_path_rename(&oldnd.path, old_dentry,
|
|
|
|
&newnd.path, new_dentry);
|
|
|
|
if (error)
|
|
|
|
goto exit6;
|
2005-04-17 05:20:36 +07:00
|
|
|
error = vfs_rename(old_dir->d_inode, old_dentry,
|
|
|
|
new_dir->d_inode, new_dentry);
|
2008-12-17 11:24:15 +07:00
|
|
|
exit6:
|
2008-02-16 05:37:49 +07:00
|
|
|
mnt_drop_write(oldnd.path.mnt);
|
2005-04-17 05:20:36 +07:00
|
|
|
exit5:
|
|
|
|
dput(new_dentry);
|
|
|
|
exit4:
|
|
|
|
dput(old_dentry);
|
|
|
|
exit3:
|
|
|
|
unlock_rename(new_dir, old_dir);
|
|
|
|
exit2:
|
2008-02-15 10:34:35 +07:00
|
|
|
path_put(&newnd.path);
|
2008-07-21 20:32:51 +07:00
|
|
|
putname(to);
|
2005-04-17 05:20:36 +07:00
|
|
|
exit1:
|
2008-02-15 10:34:35 +07:00
|
|
|
path_put(&oldnd.path);
|
2005-04-17 05:20:36 +07:00
|
|
|
putname(from);
|
2008-07-21 20:32:51 +07:00
|
|
|
exit:
|
2005-04-17 05:20:36 +07:00
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2009-01-14 20:14:17 +07:00
|
|
|
SYSCALL_DEFINE2(rename, const char __user *, oldname, const char __user *, newname)
|
2006-01-19 08:43:53 +07:00
|
|
|
{
|
|
|
|
return sys_renameat(AT_FDCWD, oldname, AT_FDCWD, newname);
|
|
|
|
}
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
int vfs_readlink(struct dentry *dentry, char __user *buffer, int buflen, const char *link)
|
|
|
|
{
|
|
|
|
int len;
|
|
|
|
|
|
|
|
len = PTR_ERR(link);
|
|
|
|
if (IS_ERR(link))
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
len = strlen(link);
|
|
|
|
if (len > (unsigned) buflen)
|
|
|
|
len = buflen;
|
|
|
|
if (copy_to_user(buffer, link, len))
|
|
|
|
len = -EFAULT;
|
|
|
|
out:
|
|
|
|
return len;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* A helper for ->readlink(). This should be used *ONLY* for symlinks that
|
|
|
|
* have ->follow_link() touching nd only in nd_set_link(). Using (or not
|
|
|
|
* using) it for any given inode is up to filesystem.
|
|
|
|
*/
|
|
|
|
int generic_readlink(struct dentry *dentry, char __user *buffer, int buflen)
|
|
|
|
{
|
|
|
|
struct nameidata nd;
|
2005-08-20 08:02:56 +07:00
|
|
|
void *cookie;
|
2008-06-10 06:40:37 +07:00
|
|
|
int res;
|
2005-08-20 08:02:56 +07:00
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
nd.depth = 0;
|
2005-08-20 08:02:56 +07:00
|
|
|
cookie = dentry->d_inode->i_op->follow_link(dentry, &nd);
|
2008-06-10 06:40:37 +07:00
|
|
|
if (IS_ERR(cookie))
|
|
|
|
return PTR_ERR(cookie);
|
|
|
|
|
|
|
|
res = vfs_readlink(dentry, buffer, buflen, nd_get_link(&nd));
|
|
|
|
if (dentry->d_inode->i_op->put_link)
|
|
|
|
dentry->d_inode->i_op->put_link(dentry, &nd, cookie);
|
|
|
|
return res;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
int vfs_follow_link(struct nameidata *nd, const char *link)
|
|
|
|
{
|
|
|
|
return __vfs_follow_link(nd, link);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* get the link contents into pagecache */
|
|
|
|
static char *page_getlink(struct dentry * dentry, struct page **ppage)
|
|
|
|
{
|
2008-12-20 03:47:12 +07:00
|
|
|
char *kaddr;
|
|
|
|
struct page *page;
|
2005-04-17 05:20:36 +07:00
|
|
|
struct address_space *mapping = dentry->d_inode->i_mapping;
|
2006-06-23 16:05:08 +07:00
|
|
|
page = read_mapping_page(mapping, 0, NULL);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (IS_ERR(page))
|
2007-05-07 04:49:04 +07:00
|
|
|
return (char*)page;
|
2005-04-17 05:20:36 +07:00
|
|
|
*ppage = page;
|
2008-12-20 03:47:12 +07:00
|
|
|
kaddr = kmap(page);
|
|
|
|
nd_terminate_link(kaddr, dentry->d_inode->i_size, PAGE_SIZE - 1);
|
|
|
|
return kaddr;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
int page_readlink(struct dentry *dentry, char __user *buffer, int buflen)
|
|
|
|
{
|
|
|
|
struct page *page = NULL;
|
|
|
|
char *s = page_getlink(dentry, &page);
|
|
|
|
int res = vfs_readlink(dentry,buffer,buflen,s);
|
|
|
|
if (page) {
|
|
|
|
kunmap(page);
|
|
|
|
page_cache_release(page);
|
|
|
|
}
|
|
|
|
return res;
|
|
|
|
}
|
|
|
|
|
2005-08-20 08:02:56 +07:00
|
|
|
void *page_follow_link_light(struct dentry *dentry, struct nameidata *nd)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2005-08-20 08:02:56 +07:00
|
|
|
struct page *page = NULL;
|
2005-04-17 05:20:36 +07:00
|
|
|
nd_set_link(nd, page_getlink(dentry, &page));
|
2005-08-20 08:02:56 +07:00
|
|
|
return page;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2005-08-20 08:02:56 +07:00
|
|
|
void page_put_link(struct dentry *dentry, struct nameidata *nd, void *cookie)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2005-08-20 08:02:56 +07:00
|
|
|
struct page *page = cookie;
|
|
|
|
|
|
|
|
if (page) {
|
2005-04-17 05:20:36 +07:00
|
|
|
kunmap(page);
|
|
|
|
page_cache_release(page);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
fs: symlink write_begin allocation context fix
With the write_begin/write_end aops, page_symlink was broken because it
could no longer pass a GFP_NOFS type mask into the point where the
allocations happened. They are done in write_begin, which would always
assume that the filesystem can be entered from reclaim. This bug could
cause filesystem deadlocks.
The funny thing with having a gfp_t mask there is that it doesn't really
allow the caller to arbitrarily tinker with the context in which it can be
called. It couldn't ever be GFP_ATOMIC, for example, because it needs to
take the page lock. The only thing any callers care about is __GFP_FS
anyway, so turn that into a single flag.
Add a new flag for write_begin, AOP_FLAG_NOFS. Filesystems can now act on
this flag in their write_begin function. Change __grab_cache_page to
accept a nofs argument as well, to honour that flag (while we're there,
change the name to grab_cache_page_write_begin which is more instructive
and does away with random leading underscores).
This is really a more flexible way to go in the end anyway -- if a
filesystem happens to want any extra allocations aside from the pagecache
ones in ints write_begin function, it may now use GFP_KERNEL (rather than
GFP_NOFS) for common case allocations (eg. ocfs2_alloc_write_ctxt, for a
random example).
[kosaki.motohiro@jp.fujitsu.com: fix ubifs]
[kosaki.motohiro@jp.fujitsu.com: fix fuse]
Signed-off-by: Nick Piggin <npiggin@suse.de>
Reviewed-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: <stable@kernel.org> [2.6.28.x]
Signed-off-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
[ Cleaned up the calling convention: just pass in the AOP flags
untouched to the grab_cache_page_write_begin() function. That
just simplifies everybody, and may even allow future expansion of the
logic. - Linus ]
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2009-01-05 03:00:53 +07:00
|
|
|
/*
|
|
|
|
* The nofs argument instructs pagecache_write_begin to pass AOP_FLAG_NOFS
|
|
|
|
*/
|
|
|
|
int __page_symlink(struct inode *inode, const char *symname, int len, int nofs)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
struct address_space *mapping = inode->i_mapping;
|
2006-03-11 18:27:13 +07:00
|
|
|
struct page *page;
|
2007-10-16 15:25:01 +07:00
|
|
|
void *fsdata;
|
2007-02-16 16:27:18 +07:00
|
|
|
int err;
|
2005-04-17 05:20:36 +07:00
|
|
|
char *kaddr;
|
fs: symlink write_begin allocation context fix
With the write_begin/write_end aops, page_symlink was broken because it
could no longer pass a GFP_NOFS type mask into the point where the
allocations happened. They are done in write_begin, which would always
assume that the filesystem can be entered from reclaim. This bug could
cause filesystem deadlocks.
The funny thing with having a gfp_t mask there is that it doesn't really
allow the caller to arbitrarily tinker with the context in which it can be
called. It couldn't ever be GFP_ATOMIC, for example, because it needs to
take the page lock. The only thing any callers care about is __GFP_FS
anyway, so turn that into a single flag.
Add a new flag for write_begin, AOP_FLAG_NOFS. Filesystems can now act on
this flag in their write_begin function. Change __grab_cache_page to
accept a nofs argument as well, to honour that flag (while we're there,
change the name to grab_cache_page_write_begin which is more instructive
and does away with random leading underscores).
This is really a more flexible way to go in the end anyway -- if a
filesystem happens to want any extra allocations aside from the pagecache
ones in ints write_begin function, it may now use GFP_KERNEL (rather than
GFP_NOFS) for common case allocations (eg. ocfs2_alloc_write_ctxt, for a
random example).
[kosaki.motohiro@jp.fujitsu.com: fix ubifs]
[kosaki.motohiro@jp.fujitsu.com: fix fuse]
Signed-off-by: Nick Piggin <npiggin@suse.de>
Reviewed-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: <stable@kernel.org> [2.6.28.x]
Signed-off-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
[ Cleaned up the calling convention: just pass in the AOP flags
untouched to the grab_cache_page_write_begin() function. That
just simplifies everybody, and may even allow future expansion of the
logic. - Linus ]
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2009-01-05 03:00:53 +07:00
|
|
|
unsigned int flags = AOP_FLAG_UNINTERRUPTIBLE;
|
|
|
|
if (nofs)
|
|
|
|
flags |= AOP_FLAG_NOFS;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2006-03-25 18:07:57 +07:00
|
|
|
retry:
|
2007-10-16 15:25:01 +07:00
|
|
|
err = pagecache_write_begin(NULL, mapping, 0, len-1,
|
fs: symlink write_begin allocation context fix
With the write_begin/write_end aops, page_symlink was broken because it
could no longer pass a GFP_NOFS type mask into the point where the
allocations happened. They are done in write_begin, which would always
assume that the filesystem can be entered from reclaim. This bug could
cause filesystem deadlocks.
The funny thing with having a gfp_t mask there is that it doesn't really
allow the caller to arbitrarily tinker with the context in which it can be
called. It couldn't ever be GFP_ATOMIC, for example, because it needs to
take the page lock. The only thing any callers care about is __GFP_FS
anyway, so turn that into a single flag.
Add a new flag for write_begin, AOP_FLAG_NOFS. Filesystems can now act on
this flag in their write_begin function. Change __grab_cache_page to
accept a nofs argument as well, to honour that flag (while we're there,
change the name to grab_cache_page_write_begin which is more instructive
and does away with random leading underscores).
This is really a more flexible way to go in the end anyway -- if a
filesystem happens to want any extra allocations aside from the pagecache
ones in ints write_begin function, it may now use GFP_KERNEL (rather than
GFP_NOFS) for common case allocations (eg. ocfs2_alloc_write_ctxt, for a
random example).
[kosaki.motohiro@jp.fujitsu.com: fix ubifs]
[kosaki.motohiro@jp.fujitsu.com: fix fuse]
Signed-off-by: Nick Piggin <npiggin@suse.de>
Reviewed-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: <stable@kernel.org> [2.6.28.x]
Signed-off-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
[ Cleaned up the calling convention: just pass in the AOP flags
untouched to the grab_cache_page_write_begin() function. That
just simplifies everybody, and may even allow future expansion of the
logic. - Linus ]
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2009-01-05 03:00:53 +07:00
|
|
|
flags, &page, &fsdata);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (err)
|
2007-10-16 15:25:01 +07:00
|
|
|
goto fail;
|
|
|
|
|
2011-11-25 22:14:27 +07:00
|
|
|
kaddr = kmap_atomic(page);
|
2005-04-17 05:20:36 +07:00
|
|
|
memcpy(kaddr, symname, len-1);
|
2011-11-25 22:14:27 +07:00
|
|
|
kunmap_atomic(kaddr);
|
2007-10-16 15:25:01 +07:00
|
|
|
|
|
|
|
err = pagecache_write_end(NULL, mapping, 0, len-1, len-1,
|
|
|
|
page, fsdata);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (err < 0)
|
|
|
|
goto fail;
|
2007-10-16 15:25:01 +07:00
|
|
|
if (err < len-1)
|
|
|
|
goto retry;
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
mark_inode_dirty(inode);
|
|
|
|
return 0;
|
|
|
|
fail:
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2006-03-11 18:27:13 +07:00
|
|
|
int page_symlink(struct inode *inode, const char *symname, int len)
|
|
|
|
{
|
|
|
|
return __page_symlink(inode, symname, len,
|
fs: symlink write_begin allocation context fix
With the write_begin/write_end aops, page_symlink was broken because it
could no longer pass a GFP_NOFS type mask into the point where the
allocations happened. They are done in write_begin, which would always
assume that the filesystem can be entered from reclaim. This bug could
cause filesystem deadlocks.
The funny thing with having a gfp_t mask there is that it doesn't really
allow the caller to arbitrarily tinker with the context in which it can be
called. It couldn't ever be GFP_ATOMIC, for example, because it needs to
take the page lock. The only thing any callers care about is __GFP_FS
anyway, so turn that into a single flag.
Add a new flag for write_begin, AOP_FLAG_NOFS. Filesystems can now act on
this flag in their write_begin function. Change __grab_cache_page to
accept a nofs argument as well, to honour that flag (while we're there,
change the name to grab_cache_page_write_begin which is more instructive
and does away with random leading underscores).
This is really a more flexible way to go in the end anyway -- if a
filesystem happens to want any extra allocations aside from the pagecache
ones in ints write_begin function, it may now use GFP_KERNEL (rather than
GFP_NOFS) for common case allocations (eg. ocfs2_alloc_write_ctxt, for a
random example).
[kosaki.motohiro@jp.fujitsu.com: fix ubifs]
[kosaki.motohiro@jp.fujitsu.com: fix fuse]
Signed-off-by: Nick Piggin <npiggin@suse.de>
Reviewed-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: <stable@kernel.org> [2.6.28.x]
Signed-off-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
[ Cleaned up the calling convention: just pass in the AOP flags
untouched to the grab_cache_page_write_begin() function. That
just simplifies everybody, and may even allow future expansion of the
logic. - Linus ]
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2009-01-05 03:00:53 +07:00
|
|
|
!(mapping_gfp_mask(inode->i_mapping) & __GFP_FS));
|
2006-03-11 18:27:13 +07:00
|
|
|
}
|
|
|
|
|
2007-02-12 15:55:39 +07:00
|
|
|
const struct inode_operations page_symlink_inode_operations = {
|
2005-04-17 05:20:36 +07:00
|
|
|
.readlink = generic_readlink,
|
|
|
|
.follow_link = page_follow_link_light,
|
|
|
|
.put_link = page_put_link,
|
|
|
|
};
|
|
|
|
|
2008-07-22 20:59:21 +07:00
|
|
|
EXPORT_SYMBOL(user_path_at);
|
Add a dentry op to allow processes to be held during pathwalk transit
Add a dentry op (d_manage) to permit a filesystem to hold a process and make it
sleep when it tries to transit away from one of that filesystem's directories
during a pathwalk. The operation is keyed off a new dentry flag
(DCACHE_MANAGE_TRANSIT).
The filesystem is allowed to be selective about which processes it holds and
which it permits to continue on or prohibits from transiting from each flagged
directory. This will allow autofs to hold up client processes whilst letting
its userspace daemon through to maintain the directory or the stuff behind it
or mounted upon it.
The ->d_manage() dentry operation:
int (*d_manage)(struct path *path, bool mounting_here);
takes a pointer to the directory about to be transited away from and a flag
indicating whether the transit is undertaken by do_add_mount() or
do_move_mount() skipping through a pile of filesystems mounted on a mountpoint.
It should return 0 if successful and to let the process continue on its way;
-EISDIR to prohibit the caller from skipping to overmounted filesystems or
automounting, and to use this directory; or some other error code to return to
the user.
->d_manage() is called with namespace_sem writelocked if mounting_here is true
and no other locks held, so it may sleep. However, if mounting_here is true,
it may not initiate or wait for a mount or unmount upon the parameter
directory, even if the act is actually performed by userspace.
Within fs/namei.c, follow_managed() is extended to check with d_manage() first
on each managed directory, before transiting away from it or attempting to
automount upon it.
follow_down() is renamed follow_down_one() and should only be used where the
filesystem deliberately intends to avoid management steps (e.g. autofs).
A new follow_down() is added that incorporates the loop done by all other
callers of follow_down() (do_add/move_mount(), autofs and NFSD; whilst AFS, NFS
and CIFS do use it, their use is removed by converting them to use
d_automount()). The new follow_down() calls d_manage() as appropriate. It
also takes an extra parameter to indicate if it is being called from mount code
(with namespace_sem writelocked) which it passes to d_manage(). follow_down()
ignores automount points so that it can be used to mount on them.
__follow_mount_rcu() is made to abort rcu-walk mode if it hits a directory with
DCACHE_MANAGE_TRANSIT set on the basis that we're probably going to have to
sleep. It would be possible to enter d_manage() in rcu-walk mode too, and have
that determine whether to abort or not itself. That would allow the autofs
daemon to continue on in rcu-walk mode.
Note that DCACHE_MANAGE_TRANSIT on a directory should be cleared when it isn't
required as every tranist from that directory will cause d_manage() to be
invoked. It can always be set again when necessary.
==========================
WHAT THIS MEANS FOR AUTOFS
==========================
Autofs currently uses the lookup() inode op and the d_revalidate() dentry op to
trigger the automounting of indirect mounts, and both of these can be called
with i_mutex held.
autofs knows that the i_mutex will be held by the caller in lookup(), and so
can drop it before invoking the daemon - but this isn't so for d_revalidate(),
since the lock is only held on _some_ of the code paths that call it. This
means that autofs can't risk dropping i_mutex from its d_revalidate() function
before it calls the daemon.
The bug could manifest itself as, for example, a process that's trying to
validate an automount dentry that gets made to wait because that dentry is
expired and needs cleaning up:
mkdir S ffffffff8014e05a 0 32580 24956
Call Trace:
[<ffffffff885371fd>] :autofs4:autofs4_wait+0x674/0x897
[<ffffffff80127f7d>] avc_has_perm+0x46/0x58
[<ffffffff8009fdcf>] autoremove_wake_function+0x0/0x2e
[<ffffffff88537be6>] :autofs4:autofs4_expire_wait+0x41/0x6b
[<ffffffff88535cfc>] :autofs4:autofs4_revalidate+0x91/0x149
[<ffffffff80036d96>] __lookup_hash+0xa0/0x12f
[<ffffffff80057a2f>] lookup_create+0x46/0x80
[<ffffffff800e6e31>] sys_mkdirat+0x56/0xe4
versus the automount daemon which wants to remove that dentry, but can't
because the normal process is holding the i_mutex lock:
automount D ffffffff8014e05a 0 32581 1 32561
Call Trace:
[<ffffffff80063c3f>] __mutex_lock_slowpath+0x60/0x9b
[<ffffffff8000ccf1>] do_path_lookup+0x2ca/0x2f1
[<ffffffff80063c89>] .text.lock.mutex+0xf/0x14
[<ffffffff800e6d55>] do_rmdir+0x77/0xde
[<ffffffff8005d229>] tracesys+0x71/0xe0
[<ffffffff8005d28d>] tracesys+0xd5/0xe0
which means that the system is deadlocked.
This patch allows autofs to hold up normal processes whilst the daemon goes
ahead and does things to the dentry tree behind the automouter point without
risking a deadlock as almost no locks are held in d_manage() and none in
d_automount().
Signed-off-by: David Howells <dhowells@redhat.com>
Was-Acked-by: Ian Kent <raven@themaw.net>
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2011-01-15 01:45:26 +07:00
|
|
|
EXPORT_SYMBOL(follow_down_one);
|
2005-04-17 05:20:36 +07:00
|
|
|
EXPORT_SYMBOL(follow_down);
|
|
|
|
EXPORT_SYMBOL(follow_up);
|
|
|
|
EXPORT_SYMBOL(get_write_access); /* binfmt_aout */
|
|
|
|
EXPORT_SYMBOL(getname);
|
|
|
|
EXPORT_SYMBOL(lock_rename);
|
|
|
|
EXPORT_SYMBOL(lookup_one_len);
|
|
|
|
EXPORT_SYMBOL(page_follow_link_light);
|
|
|
|
EXPORT_SYMBOL(page_put_link);
|
|
|
|
EXPORT_SYMBOL(page_readlink);
|
2006-03-11 18:27:13 +07:00
|
|
|
EXPORT_SYMBOL(__page_symlink);
|
2005-04-17 05:20:36 +07:00
|
|
|
EXPORT_SYMBOL(page_symlink);
|
|
|
|
EXPORT_SYMBOL(page_symlink_inode_operations);
|
2008-08-02 11:49:18 +07:00
|
|
|
EXPORT_SYMBOL(kern_path);
|
fs: introduce vfs_path_lookup
Stackable file systems, among others, frequently need to lookup paths or
path components starting from an arbitrary point in the namespace
(identified by a dentry and a vfsmount). Currently, such file systems use
lookup_one_len, which is frowned upon [1] as it does not pass the lookup
intent along; not passing a lookup intent, for example, can trigger BUG_ON's
when stacking on top of NFSv4.
The first patch introduces a new lookup function to allow lookup starting
from an arbitrary point in the namespace. This approach has been suggested
by Christoph Hellwig [2].
The second patch changes sunrpc to use vfs_path_lookup.
The third patch changes nfsctl.c to use vfs_path_lookup.
The fourth patch marks link_path_walk static.
The fifth, and last patch, unexports path_walk because it is no longer
unnecessary to call it directly, and using the new vfs_path_lookup is
cleaner.
For example, the following snippet of code, looks up "some/path/component"
in a directory pointed to by parent_{dentry,vfsmnt}:
err = vfs_path_lookup(parent_dentry, parent_vfsmnt,
"some/path/component", 0, &nd);
if (!err) {
/* exits */
...
/* once done, release the references */
path_release(&nd);
} else if (err == -ENOENT) {
/* doesn't exist */
} else {
/* other error */
}
VFS functions such as lookup_create can be used on the nameidata structure
to pass the create intent to the file system.
Signed-off-by: Josef 'Jeff' Sipek <jsipek@cs.sunysb.edu>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Acked-by: Christoph Hellwig <hch@lst.de>
Cc: Trond Myklebust <trond.myklebust@fys.uio.no>
Cc: Neil Brown <neilb@suse.de>
Cc: Michael Halcrow <mhalcrow@us.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2007-07-19 15:48:18 +07:00
|
|
|
EXPORT_SYMBOL(vfs_path_lookup);
|
2008-07-22 11:07:17 +07:00
|
|
|
EXPORT_SYMBOL(inode_permission);
|
2005-04-17 05:20:36 +07:00
|
|
|
EXPORT_SYMBOL(unlock_rename);
|
|
|
|
EXPORT_SYMBOL(vfs_create);
|
|
|
|
EXPORT_SYMBOL(vfs_follow_link);
|
|
|
|
EXPORT_SYMBOL(vfs_link);
|
|
|
|
EXPORT_SYMBOL(vfs_mkdir);
|
|
|
|
EXPORT_SYMBOL(vfs_mknod);
|
|
|
|
EXPORT_SYMBOL(generic_permission);
|
|
|
|
EXPORT_SYMBOL(vfs_readlink);
|
|
|
|
EXPORT_SYMBOL(vfs_rename);
|
|
|
|
EXPORT_SYMBOL(vfs_rmdir);
|
|
|
|
EXPORT_SYMBOL(vfs_symlink);
|
|
|
|
EXPORT_SYMBOL(vfs_unlink);
|
|
|
|
EXPORT_SYMBOL(dentry_unhash);
|
|
|
|
EXPORT_SYMBOL(generic_readlink);
|