Merge "Add more docs for datastore library" into main

# Datastore library

This library aims to manage datastore in a consistent way.

This library provides consistent API for data management (including backup,

restore, and metrics logging) on Android platform.

Notably, it is designed to be flexible and could be utilized for a wide range of

data store besides the settings preferences.

## Overview

A datastore is required to extend the `BackupRestoreStorage` class and implement

either `Observable` or `KeyedObservable` interface, which enforces:

In the high-level design, a persistent datastore aims to support two key

characteristics:

-   **observable**: triggers backup and metrics logging whenever data is

    changed.

-   **transferable**: offers users with a seamless experience by backing up and

    restoring data on to new devices.

-   Backup and restore: Datastore should support

More specifically, Android framework supports

[data backup](https://developer.android.com/guide/topics/data/backup) to

    preserve user experiences on a new device.

-   Observer pattern: The

preserve user experiences on a new device. And the

[observer pattern](https://en.wikipedia.org/wiki/Observer_pattern) allows to

    monitor data change in the datastore and

    -   trigger

        [BackupManager.dataChanged](https://developer.android.com/reference/android/app/backup/BackupManager#dataChanged\(\))

        automatically.

    -   track data change event to log metrics.

    -   update internal state and take action.

monitor data change.

### Backup and restore

The Android backup framework provides

Currently, the Android backup framework provides

[BackupAgentHelper](https://developer.android.com/reference/android/app/backup/BackupAgentHelper)

and

[BackupHelper](https://developer.android.com/reference/android/app/backup/BackupHelper)

to back up a datastore. However, there are several caveats when implement

`BackupHelper`:

to facilitate data backup. However, there are several caveats to consider when

implementing `BackupHelper`:

-   performBackup: The data is updated incrementally but it is not well

-   *performBackup*: The data is updated incrementally but it is not well

    documented. The `ParcelFileDescriptor` state parameters are normally ignored

    and data is updated even there is no change.

-   restoreEntity: The implementation must take care not to seek or close the

    underlying data source, nor read more than size() bytes from the stream when

    restore (see

-   *restoreEntity*: The implementation must take care not to seek or close the

    underlying data source, nor read more than `size()` bytes from the stream

    when restore (see

    [BackupDataInputStream](https://developer.android.com/reference/android/app/backup/BackupDataInputStream)).

    It is possible a `BackupHelper` prevents other `BackupHelper`s from

    restoring data.

-   writeNewStateDescription: Existing implementations rarely notice that this

    callback is invoked after all entities are restored, and check if necessary

    data are all restored in `restoreEntity` (e.g.

    It is possible that a `BackupHelper` interferes with the restore process of

    other `BackupHelper`s.

-   *writeNewStateDescription*: Existing implementations rarely notice that this

    callback is invoked after *all* entities are restored. Instead, they check

    if necessary data are all restored in the `restoreEntity` (e.g.

    [BatteryBackupHelper](https://cs.android.com/android/platform/superproject/main/+/main:packages/apps/Settings/src/com/android/settings/fuelgauge/BatteryBackupHelper.java;l=144;drc=cca804e1ed504e2d477be1e3db00fb881ca32736)),

    which is not robust sometimes.

This library provides more clear API and offers some improvements:

The datastore library will mitigate these problems by providing alternative

APIs. For instance, library users make use of `InputStream` / `OutputStream` to

back up and restore data directly.

### Observer pattern

In the current implementation, the Android backup framework requires a manual

call to

[BackupManager.dataChanged](https://developer.android.com/reference/android/app/backup/BackupManager#dataChanged\(\)).

However, it's often observed that this API call is forgotten when using

`SharedPreferences`. Additionally, there's a common need to log metrics when

data changed. To address these limitations, datastore API employed the observer

pattern.

-   The implementation only needs to focus on the `BackupRestoreEntity`

    interface. The `InputStream` of restore will ensure bounded data are read,

    and close the stream will be no-op.

-   The library computes checksum of the backup data automatically, so that

    unchanged data will not be sent to Android backup system.

### API design and advantages

Datastore must extend the `BackupRestoreStorage` class (subclass of

[BackupHelper](https://developer.android.com/reference/android/app/backup/BackupHelper)).

The data in a datastore is group by entity, which is represented by

`BackupRestoreEntity`. Basically, a datastore implementation only needs to focus

on the `BackupRestoreEntity`.

If the datastore is key-value based (e.g. `SharedPreferences`), implements the

`KeyedObservable` interface to offer fine-grained observer. Otherwise,

implements `Observable`. There are builtin thread-safe implementations of the

two interfaces (`KeyedDataObservable` / `DataObservable`). If it is Kotlin, use

delegation to simplify the code.

Keep in mind that the implementation should call `KeyedObservable.notifyChange`

/ `Observable.notifyChange` whenever internal data is changed, so that the

registered observer will be notified properly.

For `SharedPreferences` use case, leverage the `SharedPreferencesStorage`

directly. To back up other file based storage, extend the

`BackupRestoreFileStorage` class.

Here are some highlights of the library:

-   The restore `InputStream` will ensure bounded data are read, and close the

    stream is no-op. That being said, all entities are isolated.

-   Data checksum is computed automatically, unchanged data will not be sent to

    Android backup system.

-   Data compression is supported:

    -   ZIP best compression is enabled by default, no extra effort needs to be

        taken.

    successfully restored in those older versions. This is achieved by extending

    the `BackupRestoreFileStorage` class, and `BackupRestoreFileArchiver` will

    treat each file as an entity and do the backup / restore.

-   Manual `BackupManager.dataChanged` call is unnecessary now, the library will

    do the invocation (see next section).

-   Manual `BackupManager.dataChanged` call is unnecessary now, the framework

    will invoke the API automatically.

### Observer pattern

## Usages

Manual `BackupManager.dataChanged` call is required by current backup framework.

In practice, it is found that `SharedPreferences` usages foget to invoke the

API. Besides, there are common use cases to log metrics when data is changed.

Consequently, observer pattern is employed to resolve the issues.

This section provides [examples](example/ExampleStorage.kt) of datastore.

If the datastore is key-value based (e.g. `SharedPreferences`), implements the

`KeyedObservable` interface to offer fine-grained observer. Otherwise,

implements `Observable`. The library provides thread-safe implementations

(`KeyedDataObservable` / `DataObservable`), and Kotlin delegation will be

helpful.

Keep in mind that the implementation should call `KeyedObservable.notifyChange`

/ `Observable.notifyChange` whenever internal data is changed, so that the

registered observer will be notified properly.

## Usage and example

For `SharedPreferences` use case, leverage the `SharedPreferencesStorage`. To

back up other file based storage, extend the `BackupRestoreFileStorage` class.

Here is an example of customized datastore, which has a string to back up:

Here is a datastore with a string data:

```kotlin

class MyDataStore : ObservableBackupRestoreStorage() {

    // Another option is make it a StringEntity type and maintain a String field inside StringEntity

    @Volatile // backup/restore happens on Binder thread

class ExampleStorage : ObservableBackupRestoreStorage() {

  @Volatile // field is manipulated by multiple threads, synchronization might be needed

  var data: String? = null

    private set

  @AnyThread

  fun setData(data: String?) {

    this.data = data

    // call notifyChange to trigger backup and metrics logging whenever data is changed

    if (data != null) {

      notifyChange(ChangeReason.UPDATE)

    } else {

      notifyChange(ChangeReason.DELETE)

    }

  }

  override val name: String

        get() = "MyData"

    get() = "ExampleStorage"

  override fun createBackupRestoreEntities(): List<BackupRestoreEntity> =

    listOf(StringEntity("data"))

  override fun enableRestore(): Boolean {

    return true // check condition like flag, environment, etc.

  }

  override fun enableBackup(backupContext: BackupContext): Boolean {

    return true // check condition like flag, environment, etc.

  }

  @BinderThread

  private inner class StringEntity(override val key: String) : BackupRestoreEntity {

        override fun backup(

            backupContext: BackupContext,

            outputStream: OutputStream,

        ) =

    override fun backup(backupContext: BackupContext, outputStream: OutputStream) =

      if (data != null) {

        outputStream.write(data!!.toByteArray(UTF_8))

        EntityBackupResult.UPDATE

      } else {

                EntityBackupResult.DELETE

        EntityBackupResult.DELETE // delete existing backup data

      }

    override fun restore(restoreContext: RestoreContext, inputStream: InputStream) {

            data = String(inputStream.readAllBytes(), UTF_8)

            // NOTE: The library will call notifyChange(ChangeReason.RESTORE) for you

      // DO NOT call setData API here, which will trigger notifyChange unexpectedly.

      // Under the hood, the datastore library will call notifyChange(ChangeReason.RESTORE)

      // later to notify observers.

      data = String(inputStream.readBytes(), UTF_8)

      // Handle restored data in onRestoreFinished() callback

    }

  }

  override fun onRestoreFinished() {

        // TODO: Update state with the restored data. Use this callback instead "restore()" in case

        //       the restore action involves several entities.

    // TODO: Update state with the restored data. Use this callback instead of "restore()" in

    //       case the restore action involves several entities.

    // NOTE: The library will call notifyChange(ChangeReason.RESTORE) for you

  }

}

```

In the application class:

And this is a datastore with key value data:

```kotlin

class ExampleKeyValueStorage :

  BackupRestoreStorage(), KeyedObservable<String> by KeyedDataObservable() {

  // thread safe data structure

  private val map = ConcurrentHashMap<String, String>()

  override val name: String

    get() = "ExampleKeyValueStorage"

  fun updateData(key: String, value: String?) {

    if (value != null) {

      map[key] = value

      notifyChange(ChangeReason.UPDATE)

    } else {

      map.remove(key)

      notifyChange(ChangeReason.DELETE)

    }

  }

  override fun createBackupRestoreEntities(): List<BackupRestoreEntity> =

    listOf(createMapBackupRestoreEntity())

  private fun createMapBackupRestoreEntity() =

    object : BackupRestoreEntity {

      override val key: String

        get() = "map"

      override fun backup(

        backupContext: BackupContext,

        outputStream: OutputStream,

      ): EntityBackupResult {

        // Use TreeMap to achieve predictable and stable order, so that data will not be

        // updated to Android backup backend if there is only order change.

        val copy = TreeMap(map)

        if (copy.isEmpty()) return EntityBackupResult.DELETE

        val dataOutputStream = DataOutputStream(outputStream)

        dataOutputStream.writeInt(copy.size)

        for ((key, value) in copy) {

          dataOutputStream.writeUTF(key)

          dataOutputStream.writeUTF(value)

        }

        return EntityBackupResult.UPDATE

      }

      override fun restore(restoreContext: RestoreContext, inputStream: InputStream) {

        val dataInputString = DataInputStream(inputStream)

        repeat(dataInputString.readInt()) {

          val key = dataInputString.readUTF()

          val value = dataInputString.readUTF()

          map[key] = value

        }

      }

    }

}

```

All the datastore should be added in the application class:

```kotlin

class MyApplication : Application() {

class ExampleApplication : Application() {

  override fun onCreate() {

    super.onCreate();

    BackupRestoreStorageManager.getInstance(this).add(MyDataStore());

    super.onCreate()

    BackupRestoreStorageManager.getInstance(this)

      .add(ExampleStorage(), ExampleKeyValueStorage())

  }

}

```

In the custom `BackupAgentHelper` class:

Additionally, inject datastore to the custom `BackupAgentHelper` class:

```kotlin

class MyBackupAgentHelper : BackupAgentHelper() {

class ExampleBackupAgent : BackupAgentHelper() {

  override fun onCreate() {

    BackupRestoreStorageManager.getInstance(this).addBackupAgentHelpers(this);

    super.onCreate()

    BackupRestoreStorageManager.getInstance(this).addBackupAgentHelpers(this)

  }

  override fun onRestoreFinished() {

    BackupRestoreStorageManager.getInstance(this).onRestoreFinished();

    BackupRestoreStorageManager.getInstance(this).onRestoreFinished()

  }

}

```

## Development

Please preserve the code coverage ratio during development. The current line

coverage is **100% (444/444)** and branch coverage is **93.6% (176/188)**.

import java.io.InputStream

import java.io.OutputStream

/** Entity for back up and restore. */

/**

 * Entity for back up and restore.

 *

 * Note that backup/restore callback is invoked on the binder thread.

 */

interface BackupRestoreEntity {

    /**

     * Key of the entity.

    /**

     * Backs up the entity.

     *

     * Back up data in predictable order (e.g. use `TreeMap` instead of `HashMap`), otherwise data

     * will be backed up needlessly.

     *

     * @param backupContext context for backup

     * @param outputStream output stream to back up data

     * @return false if backup file is deleted, otherwise true

     * @return backup result

     */

    @BinderThread

    @Throws(IOException::class)

import android.app.backup.BackupHelper

import android.os.ParcelFileDescriptor

import android.util.Log

import androidx.annotation.BinderThread

import androidx.annotation.VisibleForTesting

import androidx.collection.MutableScatterMap

import com.google.common.io.ByteStreams

import java.util.zip.CheckedInputStream

import java.util.zip.CheckedOutputStream

import java.util.zip.Checksum

import javax.annotation.concurrent.ThreadSafe

internal const val LOG_TAG = "BackupRestoreStorage"

/**

 * Storage with backup and restore support. Subclass must implement either [Observable] or

 * [KeyedObservable] interface.

 * Storage with backup and restore support.

 *

 * Subclass MUST

 * - implement either [Observable] or [KeyedObservable] interface.

 * - be thread safe, backup/restore happens on Binder thread, while general data read/write

 *   operations occur on other threads.

 *

 * The storage is identified by a unique string [name] and data set is split into entities

 * ([BackupRestoreEntity]).

 */

@ThreadSafe

abstract class BackupRestoreStorage : BackupHelper {

    /**

     * A unique string used to disambiguate the various storages within backup agent.

    @VisibleForTesting internal var entities: List<BackupRestoreEntity>? = null

    /** Entities to back up and restore. */

    abstract fun createBackupRestoreEntities(): List<BackupRestoreEntity>

    @BinderThread abstract fun createBackupRestoreEntities(): List<BackupRestoreEntity>

    /** Default codec used to encode/decode the entity data. */

    open fun defaultCodec(): BackupCodec = BackupZipCodec.BEST_COMPRESSION

        Log.i(LOG_TAG, "[$name] Backup end")

    }

    /** Returns if backup is enabled. */

    /**

     * Returns if backup is enabled.

     *

     * If disabled, [performBackup] will be no-op, all entities backup are skipped.

     */

    open fun enableBackup(backupContext: BackupContext): Boolean = true

    open fun wrapBackupOutputStream(codec: BackupCodec, outputStream: OutputStream): OutputStream {

    private fun ensureEntities(): List<BackupRestoreEntity> =

        entities ?: createBackupRestoreEntities().also { entities = it }

    /** Returns if restore is enabled. */

    /**

     * Returns if restore is enabled.

     *

     * If disabled, [restoreEntity] will be no-op, all entities restore are skipped.

     */

    open fun enableRestore(): Boolean = true

    open fun wrapRestoreInputStream(

    }

    final override fun writeNewStateDescription(newState: ParcelFileDescriptor) {

        if (!enableRestore()) return

        entities = null // clear to reduce memory footprint

        newState.writeAndClearEntityStates()

        onRestoreFinished()

    }

    /** Callbacks when restore finished. */

    /** Callbacks when entity data are all restored. */

    open fun onRestoreFinished() {}

    @VisibleForTesting

        verify(storage).onRestoreFinished()

    }

    @Test

    fun writeNewStateDescription_restoreDisabled() {

        val storage = spy(TestStorage().apply { enabled = false })

        temporaryFolder.newFile().toParcelFileDescriptor(MODE_WRITE_ONLY or MODE_APPEND).use {

            storage.writeNewStateDescription(it)

        }

        verify(storage, never()).onRestoreFinished()

    }

    @Test

    fun backupAndRestore() {

        val storage = spy(TestStorage(entity1, entity2))