Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs

The journalling layer is  easy to use. You need to

first of all create a journal_t data structure. There are

two calls to do this dependent on how you decide to allocate the physical

media on which the journal resides. The journal_init_inode() call

is for journals stored in filesystem inodes, or the journal_init_dev()

call can be use for journal stored on a raw device (in a continuous range

media on which the journal resides. The jbd2_journal_init_inode() call

is for journals stored in filesystem inodes, or the jbd2_journal_init_dev()

call can be used for journal stored on a raw device (in a continuous range

of blocks). A journal_t is a typedef for a struct pointer, so when

you are finally finished make sure you call journal_destroy() on it

you are finally finished make sure you call jbd2_journal_destroy() on it

to free up any used kernel memory.

</para>

<para>

Once you have got your journal_t object you need to 'mount' or load the journal

file, unless of course you haven't initialised it yet - in which case you

need to call journal_create().

file. The journalling layer expects the space for the journal was already

allocated and initialized properly by the userspace tools.  When loading the

journal you must call jbd2_journal_load() to process journal contents.  If the

client file system detects the journal contents does not need to be processed

(or even need not have valid contents), it may call jbd2_journal_wipe() to

clear the journal contents before calling jbd2_journal_load().

</para>

<para>

Most of the time however your journal file will already have been created, but

before you load it you must call journal_wipe() to empty the journal file.

Hang on, you say , what if the filesystem wasn't cleanly umount()'d . Well, it is the

job of the client file system to detect this and skip the call to journal_wipe().

</para>

<para>

In either case the next call should be to journal_load() which prepares the

journal file for use. Note that journal_wipe(..,0) calls journal_skip_recovery()

for you if it detects any outstanding transactions in the journal and similarly

journal_load() will call journal_recover() if necessary.

I would advise reading fs/ext3/super.c for examples on this stage.

[RGG: Why is the journal_wipe() call necessary - doesn't this needlessly

complicate the API. Or isn't a good idea for the journal layer to hide

dirty mounts from the client fs]

Note that jbd2_journal_wipe(..,0) calls jbd2_journal_skip_recovery() for you if

it detects any outstanding transactions in the journal and similarly

jbd2_journal_load() will call jbd2_journal_recover() if necessary.  I would

advise reading ext4_load_journal() in fs/ext4/super.c for examples on this

stage.

</para>

<para>

is done by wrapping them into transactions. Additionally you

also need to wrap the modification of each of the buffers

with calls to the journal layer, so it knows what the modifications

you are actually making are. To do this use  journal_start() which

you are actually making are. To do this use jbd2_journal_start() which

returns a transaction handle.

</para>

<para>

journal_start()

and its counterpart journal_stop(), which indicates the end of a transaction

are nestable calls, so you can reenter a transaction if necessary,

but remember you must call journal_stop() the same number of times as

journal_start() before the transaction is completed (or more accurately

leaves the update phase). Ext3/VFS makes use of this feature to simplify

quota support.

jbd2_journal_start()

and its counterpart jbd2_journal_stop(), which indicates the end of a

transaction are nestable calls, so you can reenter a transaction if necessary,

but remember you must call jbd2_journal_stop() the same number of times as

jbd2_journal_start() before the transaction is completed (or more accurately

leaves the update phase). Ext4/VFS makes use of this feature to simplify

handling of inode dirtying, quota support, etc.

</para>

<para>

Inside each transaction you need to wrap the modifications to the

individual buffers (blocks). Before you start to modify a buffer you

need to call journal_get_{create,write,undo}_access() as appropriate,

need to call jbd2_journal_get_{create,write,undo}_access() as appropriate,

this allows the journalling layer to copy the unmodified data if it

needs to. After all the buffer may be part of a previously uncommitted

transaction.

At this point you are at last ready to modify a buffer, and once

you are have done so you need to call journal_dirty_{meta,}data().

you are have done so you need to call jbd2_journal_dirty_{meta,}data().

Or if you've asked for access to a buffer you now know is now longer

required to be pushed back on the device you can call journal_forget()

required to be pushed back on the device you can call jbd2_journal_forget()

in much the same way as you might have used bforget() in the past.

</para>

<para>

A journal_flush() may be called at any time to commit and checkpoint

A jbd2_journal_flush() may be called at any time to commit and checkpoint

all your transactions.

</para>

<para>

Then at umount time , in your put_super() you can then call journal_destroy()

Then at umount time , in your put_super() you can then call jbd2_journal_destroy()

to clean up your in-core journal object.

</para>

Unfortunately there a couple of ways the journal layer can cause a deadlock.

The first thing to note is that each task can only have

a single outstanding transaction at any one time, remember nothing

commits until the outermost journal_stop(). This means

commits until the outermost jbd2_journal_stop(). This means

you must complete the transaction at the end of each file/inode/address

etc. operation you perform, so that the journalling system isn't re-entered

on another journal. Since transactions can't be nested/batched

across differing journals, and another filesystem other than

yours (say ext3) may be modified in a later syscall.

yours (say ext4) may be modified in a later syscall.

</para>

<para>

The second case to bear in mind is that journal_start() can

The second case to bear in mind is that jbd2_journal_start() can

block if there isn't enough space in the journal for your transaction

(based on the passed nblocks param) - when it blocks it merely(!) needs to

wait for transactions to complete and be committed from other tasks,

so essentially we are waiting for journal_stop(). So to avoid

deadlocks you must treat journal_start/stop() as if they

so essentially we are waiting for jbd2_journal_stop(). So to avoid

deadlocks you must treat jbd2_journal_start/stop() as if they

were semaphores and include them in your semaphore ordering rules to prevent

deadlocks. Note that journal_extend() has similar blocking behaviour to

journal_start() so you can deadlock here just as easily as on journal_start().

deadlocks. Note that jbd2_journal_extend() has similar blocking behaviour to

jbd2_journal_start() so you can deadlock here just as easily as on

jbd2_journal_start().

</para>

<para>

Try to reserve the right number of blocks the first time. ;-). This will

be the maximum number of blocks you are going to touch in this transaction.

I advise having a look at at least ext3_jbd.h to see the basis on which

ext3 uses to make these decisions.

I advise having a look at at least ext4_jbd.h to see the basis on which

ext4 uses to make these decisions.

</para>

<para>

Another wriggle to watch out for is your on-disk block allocation strategy.

why? Because, if you undo a delete, you need to ensure you haven't reused any

of the freed blocks in a later transaction. One simple way of doing this

is make sure any blocks you allocate only have checkpointed transactions

listed against them. Ext3 does this in ext3_test_allocatable().

Why? Because, if you do a delete, you need to ensure you haven't reused any

of the freed blocks until the transaction freeing these blocks commits. If you

reused these blocks and crash happens, there is no way to restore the contents

of the reallocated blocks at the end of the last fully committed transaction.

One simple way of doing this is to mark blocks as free in internal in-memory

block allocation structures only after the transaction freeing them commits.

Ext4 uses journal commit callback for this purpose.

</para>

<para>

With journal commit callbacks you can ask the journalling layer to call a

callback function when the transaction is finally committed to disk, so that

you can do some of your own management. You ask the journalling layer for

calling the callback by simply setting journal->j_commit_callback function

pointer and that function is called after each transaction commit. You can also

use transaction->t_private_list for attaching entries to a transaction that

need processing when the transaction commits.

</para>

<para>

Lock is also providing through journal_{un,}lock_updates(),

ext3 uses this when it wants a window with a clean and stable fs for a moment.

eg.

JBD2 also provides a way to block all transaction updates via

jbd2_journal_{un,}lock_updates(). Ext4 uses this when it wants a window with a

clean and stable fs for a moment.  E.g.

</para>

<programlisting>

	journal_lock_updates() //stop new stuff happening..

	journal_flush()        // checkpoint everything.

	jbd2_journal_lock_updates() //stop new stuff happening..

	jbd2_journal_flush()        // checkpoint everything.

	..do stuff on stable fs

	journal_unlock_updates() // carry on with filesystem use.

	jbd2_journal_unlock_updates() // carry on with filesystem use.

</programlisting>

<para>

The opportunities for abuse and DOS attacks with this should be obvious,

if you allow unprivileged userspace to trigger codepaths containing these

calls.

</para>

<para>

A new feature of jbd since 2.5.25 is commit callbacks with the new

journal_callback_set() function you can now ask the journalling layer

to call you back when the transaction is finally committed to disk, so that

you can do some of your own management. The key to this is the journal_callback

struct, this maintains the internal callback information but you can

extend it like this:-

</para>

<programlisting>

	struct  myfs_callback_s {

		//Data structure element required by jbd..

		struct journal_callback for_jbd;

		// Stuff for myfs allocated together.

		myfs_inode*    i_commited;

	}

</programlisting>

<para>

this would be useful if you needed to know when data was committed to a

particular inode.

</para>

    </sect2>

to tell the journalling layer about them.

</para>

<para>

Here is a some pseudo code to give you an idea of how it works, as

an example.

</para>

<programlisting>

  journal_t* my_jnrl = journal_create();

  journal_init_{dev,inode}(jnrl,...)

  if (clean) journal_wipe();

  journal_load();

   foreach(transaction) { /*transactions must be

                            completed before

                            a syscall returns to

                            userspace*/

          handle_t * xct=journal_start(my_jnrl);

          foreach(bh) {

                journal_get_{create,write,undo}_access(xact,bh);

                if ( myfs_modify(bh) ) { /* returns true

                                        if makes changes */

                           journal_dirty_{meta,}data(xact,bh);

                } else {

                           journal_forget(bh);

                }

          }

          journal_stop(xct);

   }

   journal_destroy(my_jrnl);

</programlisting>

    </sect2>

    </sect1>

     <title>Data Types</title>

     <para>

	The journalling layer uses typedefs to 'hide' the concrete definitions

	of the structures used. As a client of the JBD layer you can

	of the structures used. As a client of the JBD2 layer you can

	just rely on the using the pointer as a magic cookie  of some sort.

	Obviously the hiding is not enforced as this is 'C'.

     </para>

	<sect2 id="structures"><title>Structures</title>

!Iinclude/linux/jbd.h

!Iinclude/linux/jbd2.h

	</sect2>

    </sect1>

	manage transactions

     </para>

	<sect2 id="journal_level"><title>Journal Level</title>

!Efs/jbd/journal.c

!Ifs/jbd/recovery.c

!Efs/jbd2/journal.c

!Ifs/jbd2/recovery.c

	</sect2>

	<sect2 id="transaction_level"><title>Transasction Level</title>

!Efs/jbd/transaction.c

!Efs/jbd2/transaction.c

	</sect2>

    </sect1>

    <sect1 id="see_also">

the time of the crash, then there is no guarantee of consistency for

the blocks in that transaction so they are discarded (which means any

filesystem changes they represent are also lost).

Check Documentation/filesystems/ext3.txt if you want to read more about

ext3 and journaling.

Check Documentation/filesystems/ext4.txt if you want to read more about

ext4 and journaling.

References

==========

for the 2.2 branch, and ported to 2.4 kernels by Peter Braam, Andreas Dilger,

Andrew Morton, Alexander Viro, Ted Ts'o and Stephen Tweedie.

Ext3 is the ext2 filesystem enhanced with journalling capabilities.

Ext3 is the ext2 filesystem enhanced with journalling capabilities. The

filesystem is a subset of ext4 filesystem so use ext4 driver for accessing

ext3 filesystems.

Options

=======

When mounting an ext3 filesystem, the following option are accepted:

(*) == default

ro			Mount filesystem read only. Note that ext3 will replay

			the journal (and thus write to the partition) even when

			mounted "read only". Mount options "ro,noload" can be

			used to prevent writes to the filesystem.

journal=update		Update the ext3 file system's journal to the current

			format.

journal=inum		When a journal already exists, this option is ignored.

			Otherwise, it specifies the number of the inode which

			will represent the ext3 file system's journal file.

journal_path=path

journal_dev=devnum	When the external journal device's major/minor numbers

			have changed, these options allow the user to specify

			the new journal location.  The journal device is

			identified through either its new major/minor numbers

			encoded in devnum, or via a path to the device.

norecovery		Don't load the journal on mounting. Note that this forces

noload			mount of inconsistent filesystem, which can lead to

			various problems.

data=journal		All data are committed into the journal prior to being

			written into the main file system.

data=ordered	(*)	All data are forced directly out to the main file

			system prior to its metadata being committed to the

			journal.

data=writeback		Data ordering is not preserved, data may be written

			into the main file system after its metadata has been

			committed to the journal.

commit=nrsec	(*)	Ext3 can be told to sync all its data and metadata

			every 'nrsec' seconds. The default value is 5 seconds.

			This means that if you lose your power, you will lose

			as much as the latest 5 seconds of work (your

			filesystem will not be damaged though, thanks to the

			journaling).  This default value (or any low value)

			will hurt performance, but it's good for data-safety.

			Setting it to 0 will have the same effect as leaving

			it at the default (5 seconds).

			Setting it to very large values will improve

			performance.

barrier=<0|1(*)>	This enables/disables the use of write barriers in

barrier	(*)		the jbd code.  barrier=0 disables, barrier=1 enables.

nobarrier		This also requires an IO stack which can support

			barriers, and if jbd gets an error on a barrier

			write, it will disable again with a warning.

			Write barriers enforce proper on-disk ordering

			of journal commits, making volatile disk write caches

			safe to use, at some performance penalty.  If

			your disks are battery-backed in one way or another,

			disabling barriers may safely improve performance.

			The mount options "barrier" and "nobarrier" can

			also be used to enable or disable barriers, for

			consistency with other ext3 mount options.

user_xattr		Enables Extended User Attributes.  Additionally, you

			need to have extended attribute support enabled in the

			kernel configuration (CONFIG_EXT3_FS_XATTR).  See the

			attr(5) manual page and http://acl.bestbits.at/ to

			learn more about extended attributes.

nouser_xattr		Disables Extended User Attributes.

acl			Enables POSIX Access Control Lists support.

			Additionally, you need to have ACL support enabled in

			the kernel configuration (CONFIG_EXT3_FS_POSIX_ACL).

			See the acl(5) manual page and http://acl.bestbits.at/

			for more information.

noacl			This option disables POSIX Access Control List

			support.

reservation

noreservation

bsddf 		(*)	Make 'df' act like BSD.

minixdf			Make 'df' act like Minix.

check=none		Don't do extra checking of bitmaps on mount.

nocheck

debug			Extra debugging information is sent to syslog.

errors=remount-ro	Remount the filesystem read-only on an error.

errors=continue		Keep going on a filesystem error.

errors=panic		Panic and halt the machine if an error occurs.

			(These mount options override the errors behavior

			specified in the superblock, which can be

			configured using tune2fs.)

data_err=ignore(*)	Just print an error message if an error occurs

			in a file data buffer in ordered mode.

data_err=abort		Abort the journal if an error occurs in a file

			data buffer in ordered mode.

grpid			Give objects the same group ID as their creator.

bsdgroups

nogrpid		(*)	New objects have the group ID of their creator.

sysvgroups

resgid=n		The group ID which may use the reserved blocks.

resuid=n		The user ID which may use the reserved blocks.

sb=n			Use alternate superblock at this location.

quota			These options are ignored by the filesystem. They

noquota			are used only by quota tools to recognize volumes

grpquota		where quota should be turned on. See documentation

usrquota		in the quota-tools package for more details

			(http://sourceforge.net/projects/linuxquota).

jqfmt=<quota type>	These options tell filesystem details about quota

usrjquota=<file>	so that quota information can be properly updated

grpjquota=<file>	during journal replay. They replace the above

			quota options. See documentation in the quota-tools

			package for more details

			(http://sourceforge.net/projects/linuxquota).

Specification

=============

Ext3 shares all disk implementation with the ext2 filesystem, and adds

transactions capabilities to ext2.  Journaling is done by the Journaling Block

Device layer.

Journaling Block Device layer

-----------------------------

The Journaling Block Device layer (JBD) isn't ext3 specific.  It was designed

to add journaling capabilities to a block device.  The ext3 filesystem code

will inform the JBD of modifications it is performing (called a transaction).

The journal supports the transactions start and stop, and in case of a crash,

the journal can replay the transactions to quickly put the partition back into

a consistent state.

Handles represent a single atomic update to a filesystem.  JBD can handle an

external journal on a block device.

Data Mode

---------

There are 3 different data modes:

* writeback mode

In data=writeback mode, ext3 does not journal data at all.  This mode provides

a similar level of journaling as that of XFS, JFS, and ReiserFS in its default

mode - metadata journaling.  A crash+recovery can cause incorrect data to

appear in files which were written shortly before the crash.  This mode will

typically provide the best ext3 performance.

* ordered mode

In data=ordered mode, ext3 only officially journals metadata, but it logically

groups metadata and data blocks into a single unit called a transaction.  When

it's time to write the new metadata out to disk, the associated data blocks

are written first.  In general, this mode performs slightly slower than

writeback but significantly faster than journal mode.

* journal mode

data=journal mode provides full data and metadata journaling.  All new data is

written to the journal first, and then to its final location.

In the event of a crash, the journal can be replayed, bringing both data and

metadata into a consistent state.  This mode is the slowest except when data

needs to be read from and written to disk at the same time where it

outperforms all other modes.

Compatibility

-------------

Ext2 partitions can be easily convert to ext3, with `tune2fs -j <dev>`.

Ext3 is fully compatible with Ext2.  Ext3 partitions can easily be mounted as

Ext2.

External Tools

==============

See manual pages to learn more.

tune2fs: 	create a ext3 journal on a ext2 partition with the -j flag.

mke2fs: 	create a ext3 partition with the -j flag.

debugfs: 	ext2 and ext3 file system debugger.

ext2online:	online (mounted) ext2 and ext3 filesystem resizer

References

==========

kernel source:	<file:fs/ext3/>

		<file:fs/jbd/>

programs: 	http://e2fsprogs.sourceforge.net/

		http://ext2resize.sourceforge.net

useful links:	http://www.ibm.com/developerworks/library/l-fs7/index.html

        http://www.ibm.com/developerworks/library/l-fs8/index.html

	to stall to allow flushers a chance to complete some IO. Ordinarily

	it can use PageDirty and PageWriteback but some filesystems have

	more complex state (unstable pages in NFS prevent reclaim) or

	do not set those flags due to locking problems (jbd). This callback

	do not set those flags due to locking problems. This callback

	allows a filesystem to indicate to the VM if a page should be

	treated as dirty or writeback for the purposes of stalling.

F:	fs/ext2/

F:	include/linux/ext2*

EXT3 FILE SYSTEM

M:	Jan Kara <jack@suse.com>

M:	Andrew Morton <akpm@linux-foundation.org>

M:	Andreas Dilger <adilger.kernel@dilger.ca>

L:	linux-ext4@vger.kernel.org

S:	Maintained

F:	Documentation/filesystems/ext3.txt

F:	fs/ext3/

EXT4 FILE SYSTEM

M:	"Theodore Ts'o" <tytso@mit.edu>

M:	Andreas Dilger <adilger.kernel@dilger.ca>

F:	fs/jffs2/

F:	include/uapi/linux/jffs2.h

JOURNALLING LAYER FOR BLOCK DEVICES (JBD)

M:	Andrew Morton <akpm@linux-foundation.org>

M:	Jan Kara <jack@suse.com>

L:	linux-ext4@vger.kernel.org

S:	Maintained

F:	fs/jbd/

F:	include/linux/jbd.h

JOURNALLING LAYER FOR BLOCK DEVICES (JBD2)

M:	"Theodore Ts'o" <tytso@mit.edu>

M:	Jan Kara <jack@suse.com>

L:	linux-ext4@vger.kernel.org

S:	Maintained

F:	fs/jbd2/