Merge branch 'vfs-scale-working' of...

--------------------------- dentry_operations --------------------------

prototypes:

	int (*d_revalidate)(struct dentry *, int);

	int (*d_hash) (struct dentry *, struct qstr *);

	int (*d_compare) (struct dentry *, struct qstr *, struct qstr *);

	int (*d_revalidate)(struct dentry *, struct nameidata *);

	int (*d_hash)(const struct dentry *, const struct inode *,

			struct qstr *);

	int (*d_compare)(const struct dentry *, const struct inode *,

			const struct dentry *, const struct inode *,

			unsigned int, const char *, const struct qstr *);

	int (*d_delete)(struct dentry *);

	void (*d_release)(struct dentry *);

	void (*d_iput)(struct dentry *, struct inode *);

	char *(*d_dname)((struct dentry *dentry, char *buffer, int buflen);

locking rules:

		dcache_lock	rename_lock	->d_lock	may block

d_revalidate:	no		no		no		yes

d_hash		no		no		no		yes

d_compare:	no		yes		no		no 

d_delete:	yes		no		yes		no

d_release:	no		no		no		yes

d_iput:		no		no		no		yes

		rename_lock	->d_lock	may block	rcu-walk

d_revalidate:	no		no		yes (ref-walk)	maybe

d_hash		no		no		no		maybe

d_compare:	yes		no		no		maybe

d_delete:	no		yes		no		no

d_release:	no		no		yes		no

d_iput:		no		no		yes		no

d_dname:	no		no		no		no

--------------------------- inode_operations --------------------------- 

	void * (*follow_link) (struct dentry *, struct nameidata *);

	void (*put_link) (struct dentry *, struct nameidata *, void *);

	void (*truncate) (struct inode *);

	int (*permission) (struct inode *, int, struct nameidata *);

	int (*check_acl)(struct inode *, int);

	int (*permission) (struct inode *, int, unsigned int);

	int (*check_acl)(struct inode *, int, unsigned int);

	int (*setattr) (struct dentry *, struct iattr *);

	int (*getattr) (struct vfsmount *, struct dentry *, struct kstat *);

	int (*setxattr) (struct dentry *, const char *,const void *,size_t,int);

put_link:	no

truncate:	yes		(see below)

setattr:	yes

permission:	no

permission:	no (may not block if called in rcu-walk mode)

check_acl:	no

getattr:	no

setxattr:	yes

RCU-based dcache locking model

==============================

On many workloads, the most common operation on dcache is to look up a

dentry, given a parent dentry and the name of the child. Typically,

for every open(), stat() etc., the dentry corresponding to the

pathname will be looked up by walking the tree starting with the first

component of the pathname and using that dentry along with the next

component to look up the next level and so on. Since it is a frequent

operation for workloads like multiuser environments and web servers,

it is important to optimize this path.

Prior to 2.5.10, dcache_lock was acquired in d_lookup and thus in

every component during path look-up. Since 2.5.10 onwards, fast-walk

algorithm changed this by holding the dcache_lock at the beginning and

walking as many cached path component dentries as possible. This

significantly decreases the number of acquisition of

dcache_lock. However it also increases the lock hold time

significantly and affects performance in large SMP machines. Since

2.5.62 kernel, dcache has been using a new locking model that uses RCU

to make dcache look-up lock-free.

The current dcache locking model is not very different from the

existing dcache locking model. Prior to 2.5.62 kernel, dcache_lock

protected the hash chain, d_child, d_alias, d_lru lists as well as

d_inode and several other things like mount look-up. RCU-based changes

affect only the way the hash chain is protected. For everything else

the dcache_lock must be taken for both traversing as well as

updating. The hash chain updates too take the dcache_lock.  The

significant change is the way d_lookup traverses the hash chain, it

doesn't acquire the dcache_lock for this and rely on RCU to ensure

that the dentry has not been *freed*.

Dcache locking details

======================

For many multi-user workloads, open() and stat() on files are very

frequently occurring operations. Both involve walking of path names to

find the dentry corresponding to the concerned file. In 2.4 kernel,

dcache_lock was held during look-up of each path component. Contention

and cache-line bouncing of this global lock caused significant

scalability problems. With the introduction of RCU in Linux kernel,

this was worked around by making the look-up of path components during

path walking lock-free.

Safe lock-free look-up of dcache hash table

===========================================

Dcache is a complex data structure with the hash table entries also

linked together in other lists. In 2.4 kernel, dcache_lock protected

all the lists. We applied RCU only on hash chain walking. The rest of

the lists are still protected by dcache_lock.  Some of the important

changes are :

1. The deletion from hash chain is done using hlist_del_rcu() macro

   which doesn't initialize next pointer of the deleted dentry and

   this allows us to walk safely lock-free while a deletion is

   happening.

2. Insertion of a dentry into the hash table is done using

   hlist_add_head_rcu() which take care of ordering the writes - the

   writes to the dentry must be visible before the dentry is

   inserted. This works in conjunction with hlist_for_each_rcu(),

   which has since been replaced by hlist_for_each_entry_rcu(), while

   walking the hash chain. The only requirement is that all

   initialization to the dentry must be done before

   hlist_add_head_rcu() since we don't have dcache_lock protection

   while traversing the hash chain. This isn't different from the

   existing code.

3. The dentry looked up without holding dcache_lock by cannot be

   returned for walking if it is unhashed. It then may have a NULL

   d_inode or other bogosity since RCU doesn't protect the other

   fields in the dentry. We therefore use a flag DCACHE_UNHASHED to

   indicate unhashed dentries and use this in conjunction with a

   per-dentry lock (d_lock). Once looked up without the dcache_lock,

   we acquire the per-dentry lock (d_lock) and check if the dentry is

   unhashed. If so, the look-up is failed. If not, the reference count

   of the dentry is increased and the dentry is returned.

4. Once a dentry is looked up, it must be ensured during the path walk

   for that component it doesn't go away. In pre-2.5.10 code, this was

   done holding a reference to the dentry. dcache_rcu does the same.

   In some sense, dcache_rcu path walking looks like the pre-2.5.10

   version.

5. All dentry hash chain updates must take the dcache_lock as well as

   the per-dentry lock in that order. dput() does this to ensure that

   a dentry that has just been looked up in another CPU doesn't get

   deleted before dget() can be done on it.

6. There are several ways to do reference counting of RCU protected

   objects. One such example is in ipv4 route cache where deferred

   freeing (using call_rcu()) is done as soon as the reference count

   goes to zero. This cannot be done in the case of dentries because

   tearing down of dentries require blocking (dentry_iput()) which

   isn't supported from RCU callbacks. Instead, tearing down of

   dentries happen synchronously in dput(), but actual freeing happens

   later when RCU grace period is over. This allows safe lock-free

   walking of the hash chains, but a matched dentry may have been

   partially torn down. The checking of DCACHE_UNHASHED flag with

   d_lock held detects such dentries and prevents them from being

   returned from look-up.

Maintaining POSIX rename semantics

==================================

Since look-up of dentries is lock-free, it can race against a

concurrent rename operation. For example, during rename of file A to

B, look-up of either A or B must succeed.  So, if look-up of B happens

after A has been removed from the hash chain but not added to the new

hash chain, it may fail.  Also, a comparison while the name is being

written concurrently by a rename may result in false positive matches

violating rename semantics.  Issues related to race with rename are

handled as described below :

1. Look-up can be done in two ways - d_lookup() which is safe from

   simultaneous renames and __d_lookup() which is not.  If

   __d_lookup() fails, it must be followed up by a d_lookup() to

   correctly determine whether a dentry is in the hash table or

   not. d_lookup() protects look-ups using a sequence lock

   (rename_lock).

2. The name associated with a dentry (d_name) may be changed if a

   rename is allowed to happen simultaneously. To avoid memcmp() in

   __d_lookup() go out of bounds due to a rename and false positive

   comparison, the name comparison is done while holding the

   per-dentry lock. This prevents concurrent renames during this

   operation.

3. Hash table walking during look-up may move to a different bucket as

   the current dentry is moved to a different bucket due to rename.

   But we use hlists in dcache hash table and they are

   null-terminated.  So, even if a dentry moves to a different bucket,

   hash chain walk will terminate. [with a list_head list, it may not

   since termination is when the list_head in the original bucket is

   reached].  Since we redo the d_parent check and compare name while

   holding d_lock, lock-free look-up will not race against d_move().

4. There can be a theoretical race when a dentry keeps coming back to

   original bucket due to double moves. Due to this look-up may

   consider that it has never moved and can end up in a infinite loop.

   But this is not any worse that theoretical livelocks we already

   have in the kernel.

Important guidelines for filesystem developers related to dcache_rcu

====================================================================

1. Existing dcache interfaces (pre-2.5.62) exported to filesystem

   don't change. Only dcache internal implementation changes. However

   filesystems *must not* delete from the dentry hash chains directly

   using the list macros like allowed earlier. They must use dcache

   APIs like d_drop() or __d_drop() depending on the situation.

2. d_flags is now protected by a per-dentry lock (d_lock). All access

   to d_flags must be protected by it.

3. For a hashed dentry, checking of d_count needs to be protected by

   d_lock.

Papers and other documentation on dcache locking

================================================

1. Scaling dcache with RCU (http://linuxjournal.com/article.php?sid=7124).

2. http://lse.sourceforge.net/locking/dcache/dcache.html

->d_parent changes are not protected by BKL anymore.  Read access is safe

if at least one of the following is true:

	* filesystem has no cross-directory rename()

	* dcache_lock is held

	* we know that parent had been locked (e.g. we are looking at

->d_parent of ->lookup() argument).

	* we are called from ->rename().

may happen while the inode is in the middle of ->write_inode(); e.g. if you blindly

free the on-disk inode, you may end up doing that while ->write_inode() is writing

to it.

---

[mandatory]

	.d_delete() now only advises the dcache as to whether or not to cache

unreferenced dentries, and is now only called when the dentry refcount goes to

0. Even on 0 refcount transition, it must be able to tolerate being called 0,

1, or more times (eg. constant, idempotent).

---

[mandatory]

	.d_compare() calling convention and locking rules are significantly

changed. Read updated documentation in Documentation/filesystems/vfs.txt (and

look at examples of other filesystems) for guidance.

---

[mandatory]

	.d_hash() calling convention and locking rules are significantly

changed. Read updated documentation in Documentation/filesystems/vfs.txt (and

look at examples of other filesystems) for guidance.

---

[mandatory]

	dcache_lock is gone, replaced by fine grained locks. See fs/dcache.c

for details of what locks to replace dcache_lock with in order to protect

particular things. Most of the time, a filesystem only needs ->d_lock, which

protects *all* the dcache state of a given dentry.

--

[mandatory]

	Filesystems must RCU-free their inodes, if they can have been accessed

via rcu-walk path walk (basically, if the file can have had a path name in the

vfs namespace).

	i_dentry and i_rcu share storage in a union, and the vfs expects

i_dentry to be reinitialized before it is freed, so an:

  INIT_LIST_HEAD(&inode->i_dentry);

must be done in the RCU callback.

--

[recommended]

	vfs now tries to do path walking in "rcu-walk mode", which avoids

atomic operations and scalability hazards on dentries and inodes (see

Documentation/filesystems/path-walk.txt). d_hash and d_compare changes (above)

are examples of the changes required to support this. For more complex

filesystem callbacks, the vfs drops out of rcu-walk mode before the fs call, so

no changes are required to the filesystem. However, this is costly and loses

the benefits of rcu-walk mode. We will begin to add filesystem callbacks that

are rcu-walk aware, shown below. Filesystems should take advantage of this

where possible.

--

[mandatory]

	d_revalidate is a callback that is made on every path element (if

the filesystem provides it), which requires dropping out of rcu-walk mode. This

may now be called in rcu-walk mode (nd->flags & LOOKUP_RCU). -ECHILD should be

returned if the filesystem cannot handle rcu-walk. See

Documentation/filesystems/vfs.txt for more details.

	permission and check_acl are inode permission checks that are called

on many or all directory inodes on the way down a path walk (to check for

exec permission). These must now be rcu-walk aware (flags & IPERM_RCU). See

Documentation/filesystems/vfs.txt for more details.

        void * (*follow_link) (struct dentry *, struct nameidata *);

        void (*put_link) (struct dentry *, struct nameidata *, void *);

	void (*truncate) (struct inode *);

	int (*permission) (struct inode *, int, struct nameidata *);

	int (*permission) (struct inode *, int, unsigned int);

	int (*check_acl)(struct inode *, int, unsigned int);

	int (*setattr) (struct dentry *, struct iattr *);

	int (*getattr) (struct vfsmount *mnt, struct dentry *, struct kstat *);

	int (*setxattr) (struct dentry *, const char *,const void *,size_t,int);

  permission: called by the VFS to check for access rights on a POSIX-like

  	filesystem.

	May be called in rcu-walk mode (flags & IPERM_RCU). If in rcu-walk

	mode, the filesystem must check the permission without blocking or

	storing to the inode.

	If a situation is encountered that rcu-walk cannot handle, return

	-ECHILD and it will be called again in ref-walk mode.

  setattr: called by the VFS to set attributes for a file. This method

  	is called by chmod(2) and related system calls.

struct dentry_operations {

	int (*d_revalidate)(struct dentry *, struct nameidata *);

	int (*d_hash) (struct dentry *, struct qstr *);

	int (*d_compare) (struct dentry *, struct qstr *, struct qstr *);

	int (*d_delete)(struct dentry *);

	int (*d_hash)(const struct dentry *, const struct inode *,

			struct qstr *);

	int (*d_compare)(const struct dentry *, const struct inode *,

			const struct dentry *, const struct inode *,

			unsigned int, const char *, const struct qstr *);

	int (*d_delete)(const struct dentry *);

	void (*d_release)(struct dentry *);

	void (*d_iput)(struct dentry *, struct inode *);

	char *(*d_dname)(struct dentry *, char *, int);

	dcache. Most filesystems leave this as NULL, because all their

	dentries in the dcache are valid

  d_hash: called when the VFS adds a dentry to the hash table

	d_revalidate may be called in rcu-walk mode (nd->flags & LOOKUP_RCU).

	If in rcu-walk mode, the filesystem must revalidate the dentry without

	blocking or storing to the dentry, d_parent and d_inode should not be

	used without care (because they can go NULL), instead nd->inode should

	be used.

	If a situation is encountered that rcu-walk cannot handle, return

	-ECHILD and it will be called again in ref-walk mode.

  d_hash: called when the VFS adds a dentry to the hash table. The first

	dentry passed to d_hash is the parent directory that the name is

	to be hashed into. The inode is the dentry's inode.

	Same locking and synchronisation rules as d_compare regarding

	what is safe to dereference etc.

  d_compare: called to compare a dentry name with a given name. The first

	dentry is the parent of the dentry to be compared, the second is

	the parent's inode, then the dentry and inode (may be NULL) of the

	child dentry. len and name string are properties of the dentry to be

	compared. qstr is the name to compare it with.

	Must be constant and idempotent, and should not take locks if

	possible, and should not or store into the dentry or inodes.

	Should not dereference pointers outside the dentry or inodes without

	lots of care (eg.  d_parent, d_inode, d_name should not be used).

	However, our vfsmount is pinned, and RCU held, so the dentries and

	inodes won't disappear, neither will our sb or filesystem module.

	->i_sb and ->d_sb may be used.

  d_compare: called when a dentry should be compared with another

	It is a tricky calling convention because it needs to be called under

	"rcu-walk", ie. without any locks or references on things.

  d_delete: called when the last reference to a dentry is

	deleted. This means no-one is using the dentry, however it is

	still valid and in the dcache

  d_delete: called when the last reference to a dentry is dropped and the

	dcache is deciding whether or not to cache it. Return 1 to delete

	immediately, or 0 to cache the dentry. Default is NULL which means to

	always cache a reachable dentry. d_delete must be constant and

	idempotent.

  d_release: called when a dentry is really deallocated

	the usage count)

  dput: close a handle for a dentry (decrements the usage count). If

	the usage count drops to 0, the "d_delete" method is called

	and the dentry is placed on the unused list if the dentry is

	still in its parents hash list. Putting the dentry on the

	unused list just means that if the system needs some RAM, it

	goes through the unused list of dentries and deallocates them.

	If the dentry has already been unhashed and the usage count

	drops to 0, in this case the dentry is deallocated after the

	"d_delete" method is called

	the usage count drops to 0, and the dentry is still in its

	parent's hash, the "d_delete" method is called to check whether

	it should be cached. If it should not be cached, or if the dentry

	is not hashed, it is deleted. Otherwise cached dentries are put

	into an LRU list to be reclaimed on memory shortage.

  d_drop: this unhashes a dentry from its parents hash list. A

	subsequent call to dput() will deallocate the dentry if its