ALSA: compress offload API documentation

		compress_offload.txt

		=====================

	Pierre-Louis.Bossart <pierre-louis.bossart@linux.intel.com>

		Vinod Koul <vinod.koul@linux.intel.com>

Overview

Since its early days, the ALSA API was defined with PCM support or

constant bitrates payloads such as IEC61937 in mind. Arguments and

returned values in frames are the norm, making it a challenge to

extend the existing API to compressed data streams.

In recent years, audio digital signal processors (DSP) were integrated

in system-on-chip designs, and DSPs are also integrated in audio

codecs. Processing compressed data on such DSPs results in a dramatic

reduction of power consumption compared to host-based

processing. Support for such hardware has not been very good in Linux,

mostly because of a lack of a generic API available in the mainline

kernel.

Rather than requiring a compability break with an API change of the

ALSA PCM interface, a new 'Compressed Data' API is introduced to

provide a control and data-streaming interface for audio DSPs.

The design of this API was inspired by the 2-year experience with the

Intel Moorestown SOC, with many corrections required to upstream the

API in the mainline kernel instead of the staging tree and make it

usable by others.

Requirements

The main requirements are:

- separation between byte counts and time. Compressed formats may have

  a header per file, per frame, or no header at all. The payload size

  may vary from frame-to-frame. As a result, it is not possible to

  estimate reliably the duration of audio buffers when handling

  compressed data. Dedicated mechanisms are required to allow for

  reliable audio-video synchronization, which requires precise

  reporting of the number of samples rendered at any given time.

- Handling of multiple formats. PCM data only requires a specification

  of the sampling rate, number of channels and bits per sample. In

  contrast, compressed data comes in a variety of formats. Audio DSPs

  may also provide support for a limited number of audio encoders and

  decoders embedded in firmware, or may support more choices through

  dynamic download of libraries.

- Focus on main formats. This API provides support for the most

  popular formats used for audio and video capture and playback. It is

  likely that as audio compression technology advances, new formats

  will be added.

- Handling of multiple configurations. Even for a given format like

  AAC, some implementations may support AAC multichannel but HE-AAC

  stereo. Likewise WMA10 level M3 may require too much memory and cpu

  cycles. The new API needs to provide a generic way of listing these

  formats.

- Rendering/Grabbing only. This API does not provide any means of

  hardware acceleration, where PCM samples are provided back to

  user-space for additional processing. This API focuses instead on

  streaming compressed data to a DSP, with the assumption that the

  decoded samples are routed to a physical output or logical back-end.

 - Complexity hiding. Existing user-space multimedia frameworks all

  have existing enums/structures for each compressed format. This new

  API assumes the existence of a platform-specific compatibility layer

  to expose, translate and make use of the capabilities of the audio

  DSP, eg. Android HAL or PulseAudio sinks. By construction, regular

  applications are not supposed to make use of this API.

Design

The new API shares a number of concepts with with the PCM API for flow

control. Start, pause, resume, drain and stop commands have the same

semantics no matter what the content is.

The concept of memory ring buffer divided in a set of fragments is

borrowed from the ALSA PCM API. However, only sizes in bytes can be

specified.

Seeks/trick modes are assumed to be handled by the host.

The notion of rewinds/forwards is not supported. Data committed to the

ring buffer cannot be invalidated, except when dropping all buffers.

The Compressed Data API does not make any assumptions on how the data

is transmitted to the audio DSP. DMA transfers from main memory to an

embedded audio cluster or to a SPI interface for external DSPs are

possible. As in the ALSA PCM case, a core set of routines is exposed;

each driver implementer will have to write support for a set of

mandatory routines and possibly make use of optional ones.

The main additions are

- get_caps

This routine returns the list of audio formats supported. Querying the

codecs on a capture stream will return encoders, decoders will be

listed for playback streams.

- get_codec_caps For each codec, this routine returns a list of

capabilities. The intent is to make sure all the capabilities

correspond to valid settings, and to minimize the risks of

configuration failures. For example, for a complex codec such as AAC,

the number of channels supported may depend on a specific profile. If

the capabilities were exposed with a single descriptor, it may happen

that a specific combination of profiles/channels/formats may not be

supported. Likewise, embedded DSPs have limited memory and cpu cycles,

it is likely that some implementations make the list of capabilities

dynamic and dependent on existing workloads. In addition to codec

settings, this routine returns the minimum buffer size handled by the

implementation. This information can be a function of the DMA buffer

sizes, the number of bytes required to synchronize, etc, and can be

used by userspace to define how much needs to be written in the ring

buffer before playback can start.

- set_params

This routine sets the configuration chosen for a specific codec. The

most important field in the parameters is the codec type; in most

cases decoders will ignore other fields, while encoders will strictly

comply to the settings

- get_params

This routines returns the actual settings used by the DSP. Changes to

the settings should remain the exception.

- get_timestamp

The timestamp becomes a multiple field structure. It lists the number

of bytes transferred, the number of samples processed and the number

of samples rendered/grabbed. All these values can be used to determine

the avarage bitrate, figure out if the ring buffer needs to be

refilled or the delay due to decoding/encoding/io on the DSP.

Note that the list of codecs/profiles/modes was derived from the

OpenMAX AL specification instead of reinventing the wheel.

Modifications include:

- Addition of FLAC and IEC formats

- Merge of encoder/decoder capabilities

- Profiles/modes listed as bitmasks to make descriptors more compact

- Addition of set_params for decoders (missing in OpenMAX AL)

- Addition of AMR/AMR-WB encoding modes (missing in OpenMAX AL)

- Addition of format information for WMA

- Addition of encoding options when required (derived from OpenMAX IL)

- Addition of rateControlSupported (missing in OpenMAX AL)

Not supported:

- Support for VoIP/circuit-switched calls is not the target of this

  API. Support for dynamic bit-rate changes would require a tight

  coupling between the DSP and the host stack, limiting power savings.

- Packet-loss concealment is not supported. This would require an

  additional interface to let the decoder synthesize data when frames

  are lost during transmission. This may be added in the future.

- Volume control/routing is not handled by this API. Devices exposing a

  compressed data interface will be considered as regular ALSA devices;

  volume changes and routing information will be provided with regular

  ALSA kcontrols.

- Embedded audio effects. Such effects should be enabled in the same

  manner, no matter if the input was PCM or compressed.

- multichannel IEC encoding. Unclear if this is required.

- Encoding/decoding acceleration is not supported as mentioned

  above. It is possible to route the output of a decoder to a capture

  stream, or even implement transcoding capabilities. This routing

  would be enabled with ALSA kcontrols.

- Audio policy/resource management. This API does not provide any

  hooks to query the utilization of the audio DSP, nor any premption

  mechanisms.

- No notion of underun/overrun. Since the bytes written are compressed

  in nature and data written/read doesn't translate directly to

  rendered output in time, this does not deal with underrun/overun and

  maybe dealt in user-library

Credits:

- Mark Brown and Liam Girdwood for discussions on the need for this API

- Harsha Priya for her work on intel_sst compressed API

- Rakesh Ughreja for valuable feedback

- Sing Nallasellan, Sikkandar Madar and Prasanna Samaga for

  demonstrating and quantifying the benefits of audio offload on a

  real platform.