dm: add thin provisioning target

Introduction

============

This document descibes a collection of device-mapper targets that

between them implement thin-provisioning and snapshots.

The main highlight of this implementation, compared to the previous

implementation of snapshots, is that it allows many virtual devices to

be stored on the same data volume.  This simplifies administration and

allows the sharing of data between volumes, thus reducing disk usage.

Another significant feature is support for an arbitrary depth of

recursive snapshots (snapshots of snapshots of snapshots ...).  The

previous implementation of snapshots did this by chaining together

lookup tables, and so performance was O(depth).  This new

implementation uses a single data structure to avoid this degradation

with depth.  Fragmentation may still be an issue, however, in some

scenarios.

Metadata is stored on a separate device from data, giving the

administrator some freedom, for example to:

- Improve metadata resilience by storing metadata on a mirrored volume

  but data on a non-mirrored one.

- Improve performance by storing the metadata on SSD.

Status

======

These targets are very much still in the EXPERIMENTAL state.  Please

do not yet rely on them in production.  But do experiment and offer us

feedback.  Different use cases will have different performance

characteristics, for example due to fragmentation of the data volume.

If you find this software is not performing as expected please mail

dm-devel@redhat.com with details and we'll try our best to improve

things for you.

Userspace tools for checking and repairing the metadata are under

development.

Cookbook

========

This section describes some quick recipes for using thin provisioning.

They use the dmsetup program to control the device-mapper driver

directly.  End users will be advised to use a higher-level volume

manager such as LVM2 once support has been added.

Pool device

-----------

The pool device ties together the metadata volume and the data volume.

It maps I/O linearly to the data volume and updates the metadata via

two mechanisms:

- Function calls from the thin targets

- Device-mapper 'messages' from userspace which control the creation of new

  virtual devices amongst other things.

Setting up a fresh pool device

------------------------------

Setting up a pool device requires a valid metadata device, and a

data device.  If you do not have an existing metadata device you can

make one by zeroing the first 4k to indicate empty metadata.

    dd if=/dev/zero of=$metadata_dev bs=4096 count=1

The amount of metadata you need will vary according to how many blocks

are shared between thin devices (i.e. through snapshots).  If you have

less sharing than average you'll need a larger-than-average metadata device.

As a guide, we suggest you calculate the number of bytes to use in the

metadata device as 48 * $data_dev_size / $data_block_size but round it up

to 2MB if the answer is smaller.  The largest size supported is 16GB.

If you're creating large numbers of snapshots which are recording large

amounts of change, you may need find you need to increase this.

Reloading a pool table

----------------------

You may reload a pool's table, indeed this is how the pool is resized

if it runs out of space.  (N.B. While specifying a different metadata

device when reloading is not forbidden at the moment, things will go

wrong if it does not route I/O to exactly the same on-disk location as

previously.)

Using an existing pool device

-----------------------------

    dmsetup create pool \

	--table "0 20971520 thin-pool $metadata_dev $data_dev \

		 $data_block_size $low_water_mark"

$data_block_size gives the smallest unit of disk space that can be

allocated at a time expressed in units of 512-byte sectors.  People

primarily interested in thin provisioning may want to use a value such

as 1024 (512KB).  People doing lots of snapshotting may want a smaller value

such as 128 (64KB).  If you are not zeroing newly-allocated data,

a larger $data_block_size in the region of 256000 (128MB) is suggested.

$data_block_size must be the same for the lifetime of the

metadata device.

$low_water_mark is expressed in blocks of size $data_block_size.  If

free space on the data device drops below this level then a dm event

will be triggered which a userspace daemon should catch allowing it to

extend the pool device.  Only one such event will be sent.

Resuming a device with a new table itself triggers an event so the

userspace daemon can use this to detect a situation where a new table

already exceeds the threshold.

Thin provisioning

-----------------

i) Creating a new thinly-provisioned volume.

  To create a new thinly- provisioned volume you must send a message to an

  active pool device, /dev/mapper/pool in this example.

    dmsetup message /dev/mapper/pool 0 "create_thin 0"

  Here '0' is an identifier for the volume, a 24-bit number.  It's up

  to the caller to allocate and manage these identifiers.  If the

  identifier is already in use, the message will fail with -EEXIST.

ii) Using a thinly-provisioned volume.

  Thinly-provisioned volumes are activated using the 'thin' target:

    dmsetup create thin --table "0 2097152 thin /dev/mapper/pool 0"

  The last parameter is the identifier for the thinp device.

Internal snapshots

------------------

i) Creating an internal snapshot.

  Snapshots are created with another message to the pool.

  N.B.  If the origin device that you wish to snapshot is active, you

  must suspend it before creating the snapshot to avoid corruption.

  This is NOT enforced at the moment, so please be careful!

    dmsetup suspend /dev/mapper/thin

    dmsetup message /dev/mapper/pool 0 "create_snap 1 0"

    dmsetup resume /dev/mapper/thin

  Here '1' is the identifier for the volume, a 24-bit number.  '0' is the

  identifier for the origin device.

ii) Using an internal snapshot.

  Once created, the user doesn't have to worry about any connection

  between the origin and the snapshot.  Indeed the snapshot is no

  different from any other thinly-provisioned device and can be

  snapshotted itself via the same method.  It's perfectly legal to

  have only one of them active, and there's no ordering requirement on

  activating or removing them both.  (This differs from conventional

  device-mapper snapshots.)

  Activate it exactly the same way as any other thinly-provisioned volume:

    dmsetup create snap --table "0 2097152 thin /dev/mapper/pool 1"

Deactivation

------------

All devices using a pool must be deactivated before the pool itself

can be.

    dmsetup remove thin

    dmsetup remove snap

    dmsetup remove pool

Reference

=========

'thin-pool' target

------------------

i) Constructor

    thin-pool <metadata dev> <data dev> <data block size (sectors)> \

	      <low water mark (blocks)> [<number of feature args> [<arg>]*]

    Optional feature arguments:

    - 'skip_block_zeroing': skips the zeroing of newly-provisioned blocks.

    Data block size must be between 64KB (128 sectors) and 1GB

    (2097152 sectors) inclusive.

ii) Status

    <transaction id> <used metadata blocks>/<total metadata blocks>

    <used data blocks>/<total data blocks> <held metadata root>

    transaction id:

	A 64-bit number used by userspace to help synchronise with metadata

	from volume managers.

    used data blocks / total data blocks

	If the number of free blocks drops below the pool's low water mark a

	dm event will be sent to userspace.  This event is edge-triggered and

	it will occur only once after each resume so volume manager writers

	should register for the event and then check the target's status.

    held metadata root:

	The location, in sectors, of the metadata root that has been

	'held' for userspace read access.  '-' indicates there is no

	held root.  This feature is not yet implemented so '-' is

	always returned.

iii) Messages

    create_thin <dev id>

	Create a new thinly-provisioned device.

	<dev id> is an arbitrary unique 24-bit identifier chosen by

	the caller.

    create_snap <dev id> <origin id>

	Create a new snapshot of another thinly-provisioned device.

	<dev id> is an arbitrary unique 24-bit identifier chosen by

	the caller.

	<origin id> is the identifier of the thinly-provisioned device

	of which the new device will be a snapshot.

    delete <dev id>

	Deletes a thin device.  Irreversible.

    trim <dev id> <new size in sectors>

	Delete mappings from the end of a thin device.  Irreversible.

	You might want to use this if you're reducing the size of

	your thinly-provisioned device.  In many cases, due to the

	sharing of blocks between devices, it is not possible to

	determine in advance how much space 'trim' will release.  (In

	future a userspace tool might be able to perform this

	calculation.)

    set_transaction_id <current id> <new id>

	Userland volume managers, such as LVM, need a way to

	synchronise their external metadata with the internal metadata of the

	pool target.  The thin-pool target offers to store an

	arbitrary 64-bit transaction id and return it on the target's

	status line.  To avoid races you must provide what you think

	the current transaction id is when you change it with this

	compare-and-swap message.

'thin' target

-------------

i) Constructor

    thin <pool dev> <dev id>

    pool dev:

	the thin-pool device, e.g. /dev/mapper/my_pool or 253:0

    dev id:

	the internal device identifier of the device to be

	activated.

The pool doesn't store any size against the thin devices.  If you

load a thin target that is smaller than you've been using previously,

then you'll have no access to blocks mapped beyond the end.  If you

load a target that is bigger than before, then extra blocks will be

provisioned as and when needed.

If you wish to reduce the size of your thin device and potentially

regain some space then send the 'trim' message to the pool.

ii) Status

     <nr mapped sectors> <highest mapped sector>

	 as a cache, holding recently-read blocks in memory and performing

	 delayed writes.

source "drivers/md/persistent-data/Kconfig"

config DM_CRYPT

	tristate "Crypt target support"

	depends on BLK_DEV_DM

       ---help---

         Allow volume managers to take writable snapshots of a device.

config DM_THIN_PROVISIONING

       tristate "Thin provisioning target (EXPERIMENTAL)"

       depends on BLK_DEV_DM && EXPERIMENTAL

       select DM_PERSISTENT_DATA

       ---help---

         Provides thin provisioning and snapshots that share a data store.

config DM_DEBUG_BLOCK_STACK_TRACING

	boolean "Keep stack trace of thin provisioning block lock holders"

	depends on STACKTRACE_SUPPORT && DM_THIN_PROVISIONING

	select STACKTRACE

	---help---

	  Enable this for messages that may help debug problems with the

	  block manager locking used by thin provisioning.

	  If unsure, say N.

config DM_DEBUG_SPACE_MAPS

	boolean "Extra validation for thin provisioning space maps"

	depends on DM_THIN_PROVISIONING

	---help---

	  Enable this for messages that may help debug problems with the

	  space maps used by thin provisioning.

          If unsure, say N.

config DM_MIRROR

       tristate "Mirror target"

       depends on BLK_DEV_DM

dm-mirror-y	+= dm-raid1.o

dm-log-userspace-y \

		+= dm-log-userspace-base.o dm-log-userspace-transfer.o

dm-thin-pool-y	+= dm-thin.o dm-thin-metadata.o

md-mod-y	+= md.o bitmap.o

raid456-y	+= raid5.o

obj-$(CONFIG_DM_MULTIPATH_QL)	+= dm-queue-length.o

obj-$(CONFIG_DM_MULTIPATH_ST)	+= dm-service-time.o

obj-$(CONFIG_DM_SNAPSHOT)	+= dm-snapshot.o

obj-$(CONFIG_DM_PERSISTENT_DATA)	+= persistent-data/

obj-$(CONFIG_DM_MIRROR)		+= dm-mirror.o dm-log.o dm-region-hash.o

obj-$(CONFIG_DM_LOG_USERSPACE)	+= dm-log-userspace.o

obj-$(CONFIG_DM_ZERO)		+= dm-zero.o

obj-$(CONFIG_DM_RAID)	+= dm-raid.o

obj-$(CONFIG_DM_THIN_PROVISIONING)	+= dm-thin-pool.o

ifeq ($(CONFIG_DM_UEVENT),y)

dm-mod-objs			+= dm-uevent.o

/*

 * Copyright (C) 2010-2011 Red Hat, Inc.

 *

 * This file is released under the GPL.

 */

#ifndef DM_THIN_METADATA_H

#define DM_THIN_METADATA_H

#include "persistent-data/dm-block-manager.h"

#define THIN_METADATA_BLOCK_SIZE 4096

/*----------------------------------------------------------------*/

struct dm_pool_metadata;

struct dm_thin_device;

/*

 * Device identifier

 */

typedef uint64_t dm_thin_id;

/*

 * Reopens or creates a new, empty metadata volume.

 */

struct dm_pool_metadata *dm_pool_metadata_open(struct block_device *bdev,

					       sector_t data_block_size);

int dm_pool_metadata_close(struct dm_pool_metadata *pmd);

/*

 * Compat feature flags.  Any incompat flags beyond the ones

 * specified below will prevent use of the thin metadata.

 */

#define THIN_FEATURE_COMPAT_SUPP	  0UL

#define THIN_FEATURE_COMPAT_RO_SUPP	  0UL

#define THIN_FEATURE_INCOMPAT_SUPP	  0UL

/*

 * Device creation/deletion.

 */

int dm_pool_create_thin(struct dm_pool_metadata *pmd, dm_thin_id dev);

/*

 * An internal snapshot.

 *

 * You can only snapshot a quiesced origin i.e. one that is either

 * suspended or not instanced at all.

 */

int dm_pool_create_snap(struct dm_pool_metadata *pmd, dm_thin_id dev,

			dm_thin_id origin);

/*

 * Deletes a virtual device from the metadata.  It _is_ safe to call this

 * when that device is open.  Operations on that device will just start

 * failing.  You still need to call close() on the device.

 */

int dm_pool_delete_thin_device(struct dm_pool_metadata *pmd,

			       dm_thin_id dev);

/*

 * Commits _all_ metadata changes: device creation, deletion, mapping

 * updates.

 */

int dm_pool_commit_metadata(struct dm_pool_metadata *pmd);

/*

 * Set/get userspace transaction id.

 */

int dm_pool_set_metadata_transaction_id(struct dm_pool_metadata *pmd,

					uint64_t current_id,

					uint64_t new_id);

int dm_pool_get_metadata_transaction_id(struct dm_pool_metadata *pmd,

					uint64_t *result);

/*

 * Hold/get root for userspace transaction.

 */

int dm_pool_hold_metadata_root(struct dm_pool_metadata *pmd);

int dm_pool_get_held_metadata_root(struct dm_pool_metadata *pmd,

				   dm_block_t *result);

/*

 * Actions on a single virtual device.

 */

/*

 * Opening the same device more than once will fail with -EBUSY.

 */

int dm_pool_open_thin_device(struct dm_pool_metadata *pmd, dm_thin_id dev,

			     struct dm_thin_device **td);

int dm_pool_close_thin_device(struct dm_thin_device *td);

dm_thin_id dm_thin_dev_id(struct dm_thin_device *td);

struct dm_thin_lookup_result {

	dm_block_t block;

	int shared;

};

/*

 * Returns:

 *   -EWOULDBLOCK iff @can_block is set and would block.

 *   -ENODATA iff that mapping is not present.

 *   0 success

 */

int dm_thin_find_block(struct dm_thin_device *td, dm_block_t block,

		       int can_block, struct dm_thin_lookup_result *result);

/*

 * Obtain an unused block.

 */

int dm_pool_alloc_data_block(struct dm_pool_metadata *pmd, dm_block_t *result);

/*

 * Insert or remove block.

 */

int dm_thin_insert_block(struct dm_thin_device *td, dm_block_t block,

			 dm_block_t data_block);

int dm_thin_remove_block(struct dm_thin_device *td, dm_block_t block);

/*

 * Queries.

 */

int dm_thin_get_highest_mapped_block(struct dm_thin_device *td,

				     dm_block_t *highest_mapped);

int dm_thin_get_mapped_count(struct dm_thin_device *td, dm_block_t *result);

int dm_pool_get_free_block_count(struct dm_pool_metadata *pmd,

				 dm_block_t *result);

int dm_pool_get_free_metadata_block_count(struct dm_pool_metadata *pmd,

					  dm_block_t *result);

int dm_pool_get_metadata_dev_size(struct dm_pool_metadata *pmd,

				  dm_block_t *result);

int dm_pool_get_data_block_size(struct dm_pool_metadata *pmd, sector_t *result);

int dm_pool_get_data_dev_size(struct dm_pool_metadata *pmd, dm_block_t *result);

/*

 * Returns -ENOSPC if the new size is too small and already allocated

 * blocks would be lost.

 */

int dm_pool_resize_data_dev(struct dm_pool_metadata *pmd, dm_block_t new_size);

/*----------------------------------------------------------------*/

#endif