Merge aebb4fd3 on remote branch

.. SPDX-License-Identifier: GPL-2.0

=================

Inline Encryption

=================

Objective

=========

We want to support inline encryption (IE) in the kernel.

To allow for testing, we also want a crypto API fallback when actual

IE hardware is absent. We also want IE to work with layered devices

like dm and loopback (i.e. we want to be able to use the IE hardware

of the underlying devices if present, or else fall back to crypto API

en/decryption).

Constraints and notes

=====================

- IE hardware have a limited number of "keyslots" that can be programmed

  with an encryption context (key, algorithm, data unit size, etc.) at any time.

  One can specify a keyslot in a data request made to the device, and the

  device will en/decrypt the data using the encryption context programmed into

  that specified keyslot. When possible, we want to make multiple requests with

  the same encryption context share the same keyslot.

- We need a way for filesystems to specify an encryption context to use for

  en/decrypting a struct bio, and a device driver (like UFS) needs to be able

  to use that encryption context when it processes the bio.

- We need a way for device drivers to expose their capabilities in a unified

  way to the upper layers.

Design

======

We add a struct bio_crypt_ctx to struct bio that can represent an

encryption context, because we need to be able to pass this encryption

context from the FS layer to the device driver to act upon.

While IE hardware works on the notion of keyslots, the FS layer has no

knowledge of keyslots - it simply wants to specify an encryption context to

use while en/decrypting a bio.

We introduce a keyslot manager (KSM) that handles the translation from

encryption contexts specified by the FS to keyslots on the IE hardware.

This KSM also serves as the way IE hardware can expose their capabilities to

upper layers. The generic mode of operation is: each device driver that wants

to support IE will construct a KSM and set it up in its struct request_queue.

Upper layers that want to use IE on this device can then use this KSM in

the device's struct request_queue to translate an encryption context into

a keyslot. The presence of the KSM in the request queue shall be used to mean

that the device supports IE.

On the device driver end of the interface, the device driver needs to tell the

KSM how to actually manipulate the IE hardware in the device to do things like

programming the crypto key into the IE hardware into a particular keyslot. All

this is achieved through the :c:type:`struct keyslot_mgmt_ll_ops` that the

device driver passes to the KSM when creating it.

It uses refcounts to track which keyslots are idle (either they have no

encryption context programmed, or there are no in-flight struct bios

referencing that keyslot). When a new encryption context needs a keyslot, it

tries to find a keyslot that has already been programmed with the same

encryption context, and if there is no such keyslot, it evicts the least

recently used idle keyslot and programs the new encryption context into that

one. If no idle keyslots are available, then the caller will sleep until there

is at least one.

Blk-crypto

==========

The above is sufficient for simple cases, but does not work if there is a

need for a crypto API fallback, or if we are want to use IE with layered

devices. To these ends, we introduce blk-crypto. Blk-crypto allows us to

present a unified view of encryption to the FS (so FS only needs to specify

an encryption context and not worry about keyslots at all), and blk-crypto

can decide whether to delegate the en/decryption to IE hardware or to the

crypto API. Blk-crypto maintains an internal KSM that serves as the crypto

API fallback.

Blk-crypto needs to ensure that the encryption context is programmed into the

"correct" keyslot manager for IE. If a bio is submitted to a layered device

that eventually passes the bio down to a device that really does support IE, we

want the encryption context to be programmed into a keyslot for the KSM of the

device with IE support. However, blk-crypto does not know a priori whether a

particular device is the final device in the layering structure for a bio or

not. So in the case that a particular device does not support IE, since it is

possibly the final destination device for the bio, if the bio requires

encryption (i.e. the bio is doing a write operation), blk-crypto must fallback

to the crypto API *before* sending the bio to the device.

Blk-crypto ensures that:

- The bio's encryption context is programmed into a keyslot in the KSM of the

  request queue that the bio is being submitted to (or the crypto API fallback

  KSM if the request queue doesn't have a KSM), and that the ``bc_ksm``

  in the ``bi_crypt_context`` is set to this KSM

- That the bio has its own individual reference to the keyslot in this KSM.

  Once the bio passes through blk-crypto, its encryption context is programmed

  in some KSM. The "its own individual reference to the keyslot" ensures that

  keyslots can be released by each bio independently of other bios while

  ensuring that the bio has a valid reference to the keyslot when, for e.g., the

  crypto API fallback KSM in blk-crypto performs crypto on the device's behalf.

  The individual references are ensured by increasing the refcount for the

  keyslot in the ``bc_ksm`` when a bio with a programmed encryption

  context is cloned.

What blk-crypto does on bio submission

--------------------------------------

**Case 1:** blk-crypto is given a bio with only an encryption context that hasn't

been programmed into any keyslot in any KSM (for e.g. a bio from the FS).

  In this case, blk-crypto will program the encryption context into the KSM of the

  request queue the bio is being submitted to (and if this KSM does not exist,

  then it will program it into blk-crypto's internal KSM for crypto API

  fallback). The KSM that this encryption context was programmed into is stored

  as the ``bc_ksm`` in the bio's ``bi_crypt_context``.

**Case 2:** blk-crypto is given a bio whose encryption context has already been

programmed into a keyslot in the *crypto API fallback* KSM.

  In this case, blk-crypto does nothing; it treats the bio as not having

  specified an encryption context. Note that we cannot do here what we will do

  in Case 3 because we would have already encrypted the bio via the crypto API

  by this point.

**Case 3:** blk-crypto is given a bio whose encryption context has already been

programmed into a keyslot in some KSM (that is *not* the crypto API fallback

KSM).

  In this case, blk-crypto first releases that keyslot from that KSM and then

  treats the bio as in Case 1.

This way, when a device driver is processing a bio, it can be sure that

the bio's encryption context has been programmed into some KSM (either the

device driver's request queue's KSM, or blk-crypto's crypto API fallback KSM).

It then simply needs to check if the bio's ``bc_ksm`` is the device's

request queue's KSM. If so, then it should proceed with IE. If not, it should

simply do nothing with respect to crypto, because some other KSM (perhaps the

blk-crypto crypto API fallback KSM) is handling the en/decryption.

Blk-crypto will release the keyslot that is being held by the bio (and also

decrypt it if the bio is using the crypto API fallback KSM) once

``bio_remaining_done`` returns true for the bio.

Layered Devices

===============

Layered devices that wish to support IE need to create their own keyslot

manager for their request queue, and expose whatever functionality they choose.

When a layered device wants to pass a bio to another layer (either by

resubmitting the same bio, or by submitting a clone), it doesn't need to do

anything special because the bio (or the clone) will once again pass through

blk-crypto, which will work as described in Case 3. If a layered device wants

for some reason to do the IO by itself instead of passing it on to a child

device, but it also chose to expose IE capabilities by setting up a KSM in its

request queue, it is then responsible for en/decrypting the data itself. In

such cases, the device can choose to call the blk-crypto function

``blk_crypto_fallback_to_kernel_crypto_api`` (TODO: Not yet implemented), which will

cause the en/decryption to be done via the crypto API fallback.

Future Optimizations for layered devices

========================================

Creating a keyslot manager for the layered device uses up memory for each

keyslot, and in general, a layered device (like dm-linear) merely passes the

request on to a "child" device, so the keyslots in the layered device itself

might be completely unused. We can instead define a new type of KSM; the

"passthrough KSM", that layered devices can use to let blk-crypto know that

this layered device *will* pass the bio to some child device (and hence

through blk-crypto again, at which point blk-crypto can program the encryption

context, instead of programming it into the layered device's KSM). Again, if

the device "lies" and decides to do the IO itself instead of passing it on to

a child device, it is responsible for doing the en/decryption (and can choose

to call ``blk_crypto_fallback_to_kernel_crypto_api``). Another use case for the

"passthrough KSM" is for IE devices that want to manage their own keyslots/do

not have a limited number of keyslots.

Introduction:

=============

Storage encryption has been one of the most required feature from security

point of view. QTI based storage encryption solution uses general purpose

crypto engine. While this kind of solution provide a decent amount of

performance, it falls short as storage speed is improving significantly

continuously. To overcome performance degradation, newer chips are going to

have Inline Crypto Engine (ICE) embedded into storage device. ICE is supposed

to meet the line speed of storage devices.

Hardware Description

====================

ICE is a HW block that is embedded into storage device such as UFS/eMMC. By

default, ICE works in bypass mode i.e. ICE HW does not perform any crypto

operation on data to be processed by storage device. If required, ICE can be

configured to perform crypto operation in one direction (i.e. either encryption

or decryption) or in both direction(both encryption & decryption).

When a switch between the operation modes(plain to crypto or crypto to plain)

is desired for a particular partition, SW must complete all transactions for

that particular partition before switching the crypto mode i.e. no crypto, one

direction crypto or both direction crypto operation. Requests for other

partitions are not impacted due to crypto mode switch.

ICE HW currently supports AES128/256 bit ECB & XTS mode encryption algorithms.

Keys for crypto operations are loaded from SW. Keys are stored in a lookup

table(LUT) located inside ICE HW. Maximum of 32 keys can be loaded in ICE key

LUT. A Key inside the LUT can be referred using a key index.

SW Description

==============

ICE HW has catagorized ICE registers in 2 groups: those which can be accessed by

only secure side i.e. TZ and those which can be accessed by non-secure side such

as HLOS as well. This requires that ICE driver to be split in two pieces: one

running from TZ space and another from HLOS space.

ICE driver from TZ would configure keys as requested by HLOS side.

ICE driver on HLOS side is responsible for initialization of ICE HW.

SW Architecture Diagram

=======================

Following are all the components involved in the ICE driver for control path:

+++++++++++++++++++++++++++++++++++++++++

+               App layer               +

+++++++++++++++++++++++++++++++++++++++++

+             System layer              +

+   ++++++++         +++++++            +

+   + VOLD +         + PFM +            +

+   ++++++++         +++++++            +

+         ||         ||                 +

+         ||         ||                 +

+         \/         \/                 +

+        ++++++++++++++                 +

+        + LibQSEECom +                 +

+        ++++++++++++++                 +

+++++++++++++++++++++++++++++++++++++++++

+             Kernel                    +       +++++++++++++++++

+                                       +       +     KMS       +

+  +++++++  +++++++++++  +++++++++++    +       +++++++++++++++++

+  + ICE +  + Storage +  + QSEECom +    +       +   ICE Driver  +

+++++++++++++++++++++++++++++++++++++++++ <===> +++++++++++++++++

               ||                                    ||

               ||                                    ||

               \/                                    \/

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

+                      Storage Device                           +

+                      ++++++++++++++                           +

+                      +   ICE HW   +                           +

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Use Cases:

----------

a) Device bootup

ICE HW is detected during bootup time and corresponding probe function is

called. ICE driver parses its data from device tree node. ICE HW and storage

HW are tightly coupled. Storage device probing is dependent upon ICE device

probing. ICE driver configures all the required registers to put the ICE HW

in bypass mode.

b) Configuring keys

Currently, there are couple of use cases to configure the keys.

1) Full Disk Encryption(FDE)

System layer(VOLD) at invocation of apps layer would call libqseecom to create

the encryption key. Libqseecom calls qseecom driver to communicate with KMS

module on the secure side i.e. TZ. KMS would call ICE driver on the TZ side to

create and set the keys in ICE HW. At the end of transaction, VOLD would have

key index of key LUT where encryption key is present.

2) Per File Encryption (PFE)

Per File Manager(PFM) calls QSEECom api to create the key. PFM has a peer comp-

onent(PFT) at kernel layer which gets the corresponding key index from PFM.

Following are all the components involved in the ICE driver for data path:

+++++++++++++++++++++++++++++++++++++++++

+               App layer               +

+++++++++++++++++++++++++++++++++++++++++

+              VFS                      +

+---------------------------------------+

+         File System (EXT4)            +

+---------------------------------------+

+             Block Layer               +

+ --------------------------------------+

+                              +++++++  +

+              dm-req-crypt => + PFT +  +

+                              +++++++  +

+                                       +

+---------------------------------------+

+    +++++++++++           +++++++      +

+    + Storage +           + ICE +      +

+++++++++++++++++++++++++++++++++++++++++

+                  ||                   +

+                  || (Storage Req with +

+                  \/  ICE parameters ) +

+++++++++++++++++++++++++++++++++++++++++

+          Storage Device               +

+          ++++++++++++++               +

+          +   ICE HW   +               +

+++++++++++++++++++++++++++++++++++++++++

c) Data transaction

Once the crypto key has been configured, VOLD/PFM creates device mapping for

data partition. As part of device mapping VOLD passes key index, crypto

algorithm, mode and key length to dm layer. In case of PFE, keys are provided

by PFT as and when request is processed by dm-req-crypt. When any application

needs to read/write data, it would go through DM layer which would add crypto

information, provided by VOLD/PFT, to Request. For each Request, Storage driver

would ask ICE driver to configure crypto part of request. ICE driver extracts

crypto data from Request structure and provide it to storage driver which would

finally dispatch request to storage device.

d) Error Handling

Due to issue # 1 mentioned in "Known Issues", ICE driver does not register for

any interrupt. However, it enables sources of interrupt for ICE HW. After each

data transaction, Storage driver receives transaction completion event. As part

of event handling, storage driver calls  ICE driver to check if any of ICE

interrupt status is set. If yes, storage driver returns error to upper layer.

Error handling would be changed in future chips.

Interfaces

==========

ICE driver exposes interfaces for storage driver to :

1. Get the global instance of ICE driver

2. Get the implemented interfaces of the particular ice instance

3. Initialize the ICE HW

4. Reset the ICE HW

5. Resume/Suspend the ICE HW

6. Get the Crypto configuration for the data request for storage

7. Check if current data transaction has generated any interrupt

Driver Parameters

=================

This driver is built and statically linked into the kernel; therefore,

there are no module parameters supported by this driver.

There are no kernel command line parameters supported by this driver.

Power Management

================

ICE driver does not do power management on its own as it is part of storage

hardware. Whenever storage driver receives request for power collapse/suspend

resume, it would call ICE driver which exposes APIs for Storage HW. ICE HW

during power collapse or reset, wipes crypto configuration data. When ICE

driver receives request to resume, it would ask ICE driver on TZ side to

restore the configuration. ICE driver does not do anything as part of power

collapse or suspend event.

Interface:

==========

ICE driver exposes following APIs for storage driver to use:

int (*init)(struct platform_device *, void *, ice_success_cb, ice_error_cb);

	-- This function is invoked by storage controller during initialization of

	storage controller. Storage controller would provide success and error call

	backs which would be invoked asynchronously once ICE HW init is done.

int (*reset)(struct platform_device *);

	-- ICE HW reset as part of storage controller reset. When storage controller

	received reset command, it would call reset on ICE HW. As of now, ICE HW

	does not need to do anything as part of reset.

int (*resume)(struct platform_device *);

	-- ICE HW while going to reset, wipes all crypto keys and other data from ICE

	HW. ICE driver would reconfigure those data as part of resume operation.

int (*suspend)(struct platform_device *);

	-- This API would be called by storage driver when storage device is going to

	suspend mode. As of today, ICE driver does not do anything to handle suspend.

int (*config)(struct platform_device *, struct request* , struct ice_data_setting*);

	-- Storage driver would call this interface to get all crypto data required to

	perform crypto operation.

int (*status)(struct platform_device *);

	-- Storage driver would call this interface to check if previous data transfer

	generated any error.

Config options

==============

This driver is enabled by the kernel config option CONFIG_CRYPTO_DEV_MSM_ICE.

Dependencies

============

ICE driver depends upon corresponding ICE driver on TZ side to function

appropriately.

Known Issues

============

1. ICE HW emits 0s even if it has generated an interrupt

This issue has significant impact on how ICE interrupts are handled. Currently,

ICE driver does not register for any of the ICE interrupts but enables the

sources of interrupt. Once storage driver asks to check the status of interrupt,

it reads and clears the clear status and provide read status to storage driver.

This mechanism though not optimal but prevents filesystem curruption.

This issue has been fixed in newer chips.

2. ICE HW wipes all crypto data during power collapse

This issue necessiate that ICE driver on TZ side store the crypto material

which is not required in the case of general purpose crypto engine.

This issue has been fixed in newer chips.

Further Improvements

====================

Currently, Due to PFE use case, ICE driver is dependent upon dm-req-crypt to

provide the keys as part of request structure. This couples ICE driver with

dm-req-crypt based solution. It is under discussion to expose an IOCTL based

and registration based interface APIs from ICE driver. ICE driver would use

these two interfaces to find out if any key exists for current request. If

yes, choose the right key index received from IOCTL or registration based

APIs. If not, dont set any crypto parameter in the request.