Add more docs for datastore library (dd870223) · Commits · e / os / android_frameworks_base

packages/SettingsLib/DataStore/README.md

+194 −95

Original line number	Original line	Diff line number	Diff line
	# Datastore library		# Datastore library

	This library aims to manage datastore in a consistent way.		This library provides consistent API for data management (including backup,
			restore, and metrics logging) on Android platform.

			Notably, it is designed to be flexible and could be utilized for a wide range of
			data store besides the settings preferences.

	## Overview		## Overview

	A datastore is required to extend the `BackupRestoreStorage` class and implement		In the high-level design, a persistent datastore aims to support two key
	either `Observable` or `KeyedObservable` interface, which enforces:		characteristics:

			- observable: triggers backup and metrics logging whenever data is
			changed.
			- transferable: offers users with a seamless experience by backing up and
			restoring data on to new devices.

	- Backup and restore: Datastore should support		More specifically, Android framework supports
	[data backup](https://developer.android.com/guide/topics/data/backup) to		[data backup](https://developer.android.com/guide/topics/data/backup) to
	preserve user experiences on a new device.		preserve user experiences on a new device. And the
	- Observer pattern: The
	[observer pattern](https://en.wikipedia.org/wiki/Observer_pattern) allows to		[observer pattern](https://en.wikipedia.org/wiki/Observer_pattern) allows to
	monitor data change in the datastore and		monitor data change.
	- trigger
	[BackupManager.dataChanged](https://developer.android.com/reference/android/app/backup/BackupManager#dataChanged)
	automatically.
	- track data change event to log metrics.
	- update internal state and take action.

	### Backup and restore		### Backup and restore

	The Android backup framework provides		Currently, the Android backup framework provides
	[BackupAgentHelper](https://developer.android.com/reference/android/app/backup/BackupAgentHelper)		[BackupAgentHelper](https://developer.android.com/reference/android/app/backup/BackupAgentHelper)
	and		and
	[BackupHelper](https://developer.android.com/reference/android/app/backup/BackupHelper)		[BackupHelper](https://developer.android.com/reference/android/app/backup/BackupHelper)
	to back up a datastore. However, there are several caveats when implement		to facilitate data backup. However, there are several caveats to consider when
	`BackupHelper`:		implementing `BackupHelper`:

	- performBackup: The data is updated incrementally but it is not well		- performBackup: The data is updated incrementally but it is not well
	documented. The `ParcelFileDescriptor` state parameters are normally ignored		documented. The `ParcelFileDescriptor` state parameters are normally ignored
	and data is updated even there is no change.		and data is updated even there is no change.
	- restoreEntity: The implementation must take care not to seek or close the		- restoreEntity: The implementation must take care not to seek or close the
	underlying data source, nor read more than size() bytes from the stream when		underlying data source, nor read more than `size()` bytes from the stream
	restore (see		when restore (see
	[BackupDataInputStream](https://developer.android.com/reference/android/app/backup/BackupDataInputStream)).		[BackupDataInputStream](https://developer.android.com/reference/android/app/backup/BackupDataInputStream)).
	It is possible a `BackupHelper` prevents other `BackupHelper`s from		It is possible that a `BackupHelper` interferes with the restore process of
	restoring data.		other `BackupHelper`s.
	- writeNewStateDescription: Existing implementations rarely notice that this		- writeNewStateDescription: Existing implementations rarely notice that this
	callback is invoked after all entities are restored, and check if necessary		callback is invoked after all entities are restored. Instead, they check
	data are all restored in `restoreEntity` (e.g.		if necessary data are all restored in the `restoreEntity` (e.g.
	[BatteryBackupHelper](https://cs.android.com/android/platform/superproject/main/+/main:packages/apps/Settings/src/com/android/settings/fuelgauge/BatteryBackupHelper.java;l=144;drc=cca804e1ed504e2d477be1e3db00fb881ca32736)),		[BatteryBackupHelper](https://cs.android.com/android/platform/superproject/main/+/main:packages/apps/Settings/src/com/android/settings/fuelgauge/BatteryBackupHelper.java;l=144;drc=cca804e1ed504e2d477be1e3db00fb881ca32736)),
	which is not robust sometimes.		which is not robust sometimes.

	This library provides more clear API and offers some improvements:		The datastore library will mitigate these problems by providing alternative
			APIs. For instance, library users make use of `InputStream` / `OutputStream` to
			back up and restore data directly.

			### Observer pattern

			In the current implementation, the Android backup framework requires a manual
			call to
			[BackupManager.dataChanged](https://developer.android.com/reference/android/app/backup/BackupManager#dataChanged).
			However, it's often observed that this API call is forgotten when using
			`SharedPreferences`. Additionally, there's a common need to log metrics when
			data changed. To address these limitations, datastore API employed the observer
			pattern.

	- The implementation only needs to focus on the `BackupRestoreEntity`		### API design and advantages
	interface. The `InputStream` of restore will ensure bounded data are read,
	and close the stream will be no-op.		Datastore must extend the `BackupRestoreStorage` class (subclass of
	- The library computes checksum of the backup data automatically, so that		[BackupHelper](https://developer.android.com/reference/android/app/backup/BackupHelper)).
	unchanged data will not be sent to Android backup system.		The data in a datastore is group by entity, which is represented by
			`BackupRestoreEntity`. Basically, a datastore implementation only needs to focus
			on the `BackupRestoreEntity`.

			If the datastore is key-value based (e.g. `SharedPreferences`), implements the
			`KeyedObservable` interface to offer fine-grained observer. Otherwise,
			implements `Observable`. There are builtin thread-safe implementations of the
			two interfaces (`KeyedDataObservable` / `DataObservable`). If it is Kotlin, use
			delegation to simplify the code.

			Keep in mind that the implementation should call `KeyedObservable.notifyChange`
			/ `Observable.notifyChange` whenever internal data is changed, so that the
			registered observer will be notified properly.

			For `SharedPreferences` use case, leverage the `SharedPreferencesStorage`
			directly. To back up other file based storage, extend the
			`BackupRestoreFileStorage` class.

			Here are some highlights of the library:

			- The restore `InputStream` will ensure bounded data are read, and close the
			stream is no-op. That being said, all entities are isolated.
			- Data checksum is computed automatically, unchanged data will not be sent to
			Android backup system.
	- Data compression is supported:		- Data compression is supported:
	- ZIP best compression is enabled by default, no extra effort needs to be		- ZIP best compression is enabled by default, no extra effort needs to be
	taken.		taken.
	@@ -67,98 +105,159 @@ This library provides more clear API and offers some improvements:
	successfully restored in those older versions. This is achieved by extending		successfully restored in those older versions. This is achieved by extending
	the `BackupRestoreFileStorage` class, and `BackupRestoreFileArchiver` will		the `BackupRestoreFileStorage` class, and `BackupRestoreFileArchiver` will
	treat each file as an entity and do the backup / restore.		treat each file as an entity and do the backup / restore.
	- Manual `BackupManager.dataChanged` call is unnecessary now, the library will		- Manual `BackupManager.dataChanged` call is unnecessary now, the framework
	do the invocation (see next section).		will invoke the API automatically.

	### Observer pattern		## Usages

	Manual `BackupManager.dataChanged` call is required by current backup framework.		This section provides [examples](example/ExampleStorage.kt) of datastore.
	In practice, it is found that `SharedPreferences` usages foget to invoke the
	API. Besides, there are common use cases to log metrics when data is changed.
	Consequently, observer pattern is employed to resolve the issues.

	If the datastore is key-value based (e.g. `SharedPreferences`), implements the		Here is a datastore with a string data:
	`KeyedObservable` interface to offer fine-grained observer. Otherwise,
	implements `Observable`. The library provides thread-safe implementations
	(`KeyedDataObservable` / `DataObservable`), and Kotlin delegation will be
	helpful.

	Keep in mind that the implementation should call `KeyedObservable.notifyChange`
	/ `Observable.notifyChange` whenever internal data is changed, so that the
	registered observer will be notified properly.

	## Usage and example

	For `SharedPreferences` use case, leverage the `SharedPreferencesStorage`. To
	back up other file based storage, extend the `BackupRestoreFileStorage` class.

	Here is an example of customized datastore, which has a string to back up:

	```kotlin		```kotlin
	class MyDataStore : ObservableBackupRestoreStorage() {		class ExampleStorage : ObservableBackupRestoreStorage() {
	// Another option is make it a StringEntity type and maintain a String field inside StringEntity		@Volatile // field is manipulated by multiple threads, synchronization might be needed
	@Volatile // backup/restore happens on Binder thread
	var data: String? = null		var data: String? = null
	private set		private set

			@AnyThread
	fun setData(data: String?) {		fun setData(data: String?) {
	this.data = data		this.data = data
			// call notifyChange to trigger backup and metrics logging whenever data is changed
			if (data != null) {
	notifyChange(ChangeReason.UPDATE)		notifyChange(ChangeReason.UPDATE)
			} else {
			notifyChange(ChangeReason.DELETE)
			}
	}		}

	override val name: String		override val name: String
	get() = "MyData"		get() = "ExampleStorage"

	override fun createBackupRestoreEntities(): List<BackupRestoreEntity> =		override fun createBackupRestoreEntities(): List<BackupRestoreEntity> =
	listOf(StringEntity("data"))		listOf(StringEntity("data"))

			override fun enableRestore(): Boolean {
			return true // check condition like flag, environment, etc.
			}

			override fun enableBackup(backupContext: BackupContext): Boolean {
			return true // check condition like flag, environment, etc.
			}

			@BinderThread
	private inner class StringEntity(override val key: String) : BackupRestoreEntity {		private inner class StringEntity(override val key: String) : BackupRestoreEntity {
	override fun backup(		override fun backup(backupContext: BackupContext, outputStream: OutputStream) =
	backupContext: BackupContext,
	outputStream: OutputStream,
	) =
	if (data != null) {		if (data != null) {
	outputStream.write(data!!.toByteArray(UTF_8))		outputStream.write(data!!.toByteArray(UTF_8))
	EntityBackupResult.UPDATE		EntityBackupResult.UPDATE
	} else {		} else {
	EntityBackupResult.DELETE		EntityBackupResult.DELETE // delete existing backup data
	}		}

	override fun restore(restoreContext: RestoreContext, inputStream: InputStream) {		override fun restore(restoreContext: RestoreContext, inputStream: InputStream) {
	data = String(inputStream.readAllBytes(), UTF_8)		// DO NOT call setData API here, which will trigger notifyChange unexpectedly.
	// NOTE: The library will call notifyChange(ChangeReason.RESTORE) for you		// Under the hood, the datastore library will call notifyChange(ChangeReason.RESTORE)
			// later to notify observers.
			data = String(inputStream.readBytes(), UTF_8)
			// Handle restored data in onRestoreFinished() callback
	}		}
	}		}

	override fun onRestoreFinished() {		override fun onRestoreFinished() {
	// TODO: Update state with the restored data. Use this callback instead "restore()" in case		// TODO: Update state with the restored data. Use this callback instead of "restore()" in
	// the restore action involves several entities.		// case the restore action involves several entities.
	// NOTE: The library will call notifyChange(ChangeReason.RESTORE) for you		// NOTE: The library will call notifyChange(ChangeReason.RESTORE) for you
	}		}
	}		}
	```		```

	In the application class:		And this is a datastore with key value data:

			```kotlin
			class ExampleKeyValueStorage :
			BackupRestoreStorage(), KeyedObservable<String> by KeyedDataObservable() {
			// thread safe data structure
			private val map = ConcurrentHashMap<String, String>()

			override val name: String
			get() = "ExampleKeyValueStorage"

			fun updateData(key: String, value: String?) {
			if (value != null) {
			map[key] = value
			notifyChange(ChangeReason.UPDATE)
			} else {
			map.remove(key)
			notifyChange(ChangeReason.DELETE)
			}
			}

			override fun createBackupRestoreEntities(): List<BackupRestoreEntity> =
			listOf(createMapBackupRestoreEntity())

			private fun createMapBackupRestoreEntity() =
			object : BackupRestoreEntity {
			override val key: String
			get() = "map"

			override fun backup(
			backupContext: BackupContext,
			outputStream: OutputStream,
			): EntityBackupResult {
			// Use TreeMap to achieve predictable and stable order, so that data will not be
			// updated to Android backup backend if there is only order change.
			val copy = TreeMap(map)
			if (copy.isEmpty()) return EntityBackupResult.DELETE
			val dataOutputStream = DataOutputStream(outputStream)
			dataOutputStream.writeInt(copy.size)
			for ((key, value) in copy) {
			dataOutputStream.writeUTF(key)
			dataOutputStream.writeUTF(value)
			}
			return EntityBackupResult.UPDATE
			}

			override fun restore(restoreContext: RestoreContext, inputStream: InputStream) {
			val dataInputString = DataInputStream(inputStream)
			repeat(dataInputString.readInt()) {
			val key = dataInputString.readUTF()
			val value = dataInputString.readUTF()
			map[key] = value
			}
			}
			}
			}
			```

			All the datastore should be added in the application class:

	```kotlin		```kotlin
	class MyApplication : Application() {		class ExampleApplication : Application() {
	override fun onCreate() {		override fun onCreate() {
	super.onCreate();		super.onCreate()
	BackupRestoreStorageManager.getInstance(this).add(MyDataStore());		BackupRestoreStorageManager.getInstance(this)
			.add(ExampleStorage(), ExampleKeyValueStorage())
	}		}
	}		}
	```		```

	In the custom `BackupAgentHelper` class:		Additionally, inject datastore to the custom `BackupAgentHelper` class:

	```kotlin		```kotlin
	class MyBackupAgentHelper : BackupAgentHelper() {		class ExampleBackupAgent : BackupAgentHelper() {
	override fun onCreate() {		override fun onCreate() {
	BackupRestoreStorageManager.getInstance(this).addBackupAgentHelpers(this);		super.onCreate()
			BackupRestoreStorageManager.getInstance(this).addBackupAgentHelpers(this)
	}		}

	override fun onRestoreFinished() {		override fun onRestoreFinished() {
	BackupRestoreStorageManager.getInstance(this).onRestoreFinished();		BackupRestoreStorageManager.getInstance(this).onRestoreFinished()
	}		}
	}		}
	```		```

			## Development

			Please preserve the code coverage ratio during development. The current line
			coverage is 100% (444/444) and branch coverage is 93.6% (176/188).

packages/SettingsLib/DataStore/src/com/android/settingslib/datastore/BackupRestoreEntity.kt

+9 −2

Original line number	Original line	Diff line number	Diff line
	@@ -23,7 +23,11 @@ import java.io.IOException
	import java.io.InputStream		import java.io.InputStream
	import java.io.OutputStream		import java.io.OutputStream

	/** Entity for back up and restore. */		/**
			* Entity for back up and restore.
			*
			* Note that backup/restore callback is invoked on the binder thread.
			*/
	interface BackupRestoreEntity {		interface BackupRestoreEntity {
	/**		/**
	* Key of the entity.		* Key of the entity.
	@@ -45,9 +49,12 @@ interface BackupRestoreEntity {
	/**		/**
	* Backs up the entity.		* Backs up the entity.
	*		*
			* Back up data in predictable order (e.g. use `TreeMap` instead of `HashMap`), otherwise data
			* will be backed up needlessly.
			*
	* @param backupContext context for backup		* @param backupContext context for backup
	* @param outputStream output stream to back up data		* @param outputStream output stream to back up data
	* @return false if backup file is deleted, otherwise true		* @return backup result
	*/		*/
	@BinderThread		@BinderThread
	@Throws(IOException::class)		@Throws(IOException::class)

packages/SettingsLib/DataStore/src/com/android/settingslib/datastore/BackupRestoreStorage.kt

+22 −6

Original line number	Original line	Diff line number	Diff line
	@@ -22,6 +22,7 @@ import android.app.backup.BackupDataOutput
	import android.app.backup.BackupHelper		import android.app.backup.BackupHelper
	import android.os.ParcelFileDescriptor		import android.os.ParcelFileDescriptor
	import android.util.Log		import android.util.Log
			import androidx.annotation.BinderThread
	import androidx.annotation.VisibleForTesting		import androidx.annotation.VisibleForTesting
	import androidx.collection.MutableScatterMap		import androidx.collection.MutableScatterMap
	import com.google.common.io.ByteStreams		import com.google.common.io.ByteStreams
	@@ -38,16 +39,22 @@ import java.util.zip.CRC32
	import java.util.zip.CheckedInputStream		import java.util.zip.CheckedInputStream
	import java.util.zip.CheckedOutputStream		import java.util.zip.CheckedOutputStream
	import java.util.zip.Checksum		import java.util.zip.Checksum
			import javax.annotation.concurrent.ThreadSafe

	internal const val LOG_TAG = "BackupRestoreStorage"		internal const val LOG_TAG = "BackupRestoreStorage"

	/**		/**
	* Storage with backup and restore support. Subclass must implement either [Observable] or		* Storage with backup and restore support.
	* [KeyedObservable] interface.		*
			* Subclass MUST
			* - implement either [Observable] or [KeyedObservable] interface.
			* - be thread safe, backup/restore happens on Binder thread, while general data read/write
			* operations occur on other threads.
	*		*
	* The storage is identified by a unique string [name] and data set is split into entities		* The storage is identified by a unique string [name] and data set is split into entities
	* ([BackupRestoreEntity]).		* ([BackupRestoreEntity]).
	*/		*/
			@ThreadSafe
	abstract class BackupRestoreStorage : BackupHelper {		abstract class BackupRestoreStorage : BackupHelper {
	/**		/**
	* A unique string used to disambiguate the various storages within backup agent.		* A unique string used to disambiguate the various storages within backup agent.
	@@ -68,7 +75,7 @@ abstract class BackupRestoreStorage : BackupHelper {
	@VisibleForTesting internal var entities: List<BackupRestoreEntity>? = null		@VisibleForTesting internal var entities: List<BackupRestoreEntity>? = null

	/** Entities to back up and restore. */		/** Entities to back up and restore. */
	abstract fun createBackupRestoreEntities(): List<BackupRestoreEntity>		@BinderThread abstract fun createBackupRestoreEntities(): List<BackupRestoreEntity>

	/** Default codec used to encode/decode the entity data. */		/** Default codec used to encode/decode the entity data. */
	open fun defaultCodec(): BackupCodec = BackupZipCodec.BEST_COMPRESSION		open fun defaultCodec(): BackupCodec = BackupZipCodec.BEST_COMPRESSION
	@@ -134,7 +141,11 @@ abstract class BackupRestoreStorage : BackupHelper {
	Log.i(LOG_TAG, "[$name] Backup end")		Log.i(LOG_TAG, "[$name] Backup end")
	}		}

	/** Returns if backup is enabled. */		/**
			* Returns if backup is enabled.
			*
			* If disabled, [performBackup] will be no-op, all entities backup are skipped.
			*/
	open fun enableBackup(backupContext: BackupContext): Boolean = true		open fun enableBackup(backupContext: BackupContext): Boolean = true

	open fun wrapBackupOutputStream(codec: BackupCodec, outputStream: OutputStream): OutputStream {		open fun wrapBackupOutputStream(codec: BackupCodec, outputStream: OutputStream): OutputStream {
	@@ -172,7 +183,11 @@ abstract class BackupRestoreStorage : BackupHelper {
	private fun ensureEntities(): List<BackupRestoreEntity> =		private fun ensureEntities(): List<BackupRestoreEntity> =
	entities ?: createBackupRestoreEntities().also { entities = it }		entities ?: createBackupRestoreEntities().also { entities = it }

	/** Returns if restore is enabled. */		/**
			* Returns if restore is enabled.
			*
			* If disabled, [restoreEntity] will be no-op, all entities restore are skipped.
			*/
	open fun enableRestore(): Boolean = true		open fun enableRestore(): Boolean = true

	open fun wrapRestoreInputStream(		open fun wrapRestoreInputStream(
	@@ -188,12 +203,13 @@ abstract class BackupRestoreStorage : BackupHelper {
	}		}

	final override fun writeNewStateDescription(newState: ParcelFileDescriptor) {		final override fun writeNewStateDescription(newState: ParcelFileDescriptor) {
			if (!enableRestore()) return
	entities = null // clear to reduce memory footprint		entities = null // clear to reduce memory footprint
	newState.writeAndClearEntityStates()		newState.writeAndClearEntityStates()
	onRestoreFinished()		onRestoreFinished()
	}		}

	/** Callbacks when restore finished. */		/** Callbacks when entity data are all restored. */
	open fun onRestoreFinished() {}		open fun onRestoreFinished() {}

	@VisibleForTesting		@VisibleForTesting

packages/SettingsLib/DataStore/tests/src/com/android/settingslib/datastore/BackupRestoreStorageTest.kt

+9 −0

Original line number	Original line	Diff line number	Diff line
	@@ -247,6 +247,15 @@ class BackupRestoreStorageTest {
	verify(storage).onRestoreFinished()		verify(storage).onRestoreFinished()
	}		}

			@Test
			fun writeNewStateDescription_restoreDisabled() {
			val storage = spy(TestStorage().apply { enabled = false })
			temporaryFolder.newFile().toParcelFileDescriptor(MODE_WRITE_ONLY or MODE_APPEND).use {
			storage.writeNewStateDescription(it)
			}
			verify(storage, never()).onRestoreFinished()
			}

	@Test		@Test
	fun backupAndRestore() {		fun backupAndRestore() {
	val storage = spy(TestStorage(entity1, entity2))		val storage = spy(TestStorage(entity1, entity2))