GD: Add documentation about architecture and testing

### Why is gabeldorsche plural?

Please see this [informative video we've prepared](https://www.youtube.com/watch?v=vLRyJ0dawjM).

### Architecture

Guidelines for developing the Gabeldorsche (GD) stack

*   [Architecture](./docs/architecture/architecture.md)

*   [Style Guide](./docs/architecture/style_guide.md)

### Testing

Gabeldorsche (GD) was built with test driven development in mind. Three types of

tests are used in ensuring Gabeldorsche stack's stability, correctness and free

from regression.

If you are verifying something is glued or hooked up correctly inside the stack,

use a unit test.

*   [GTest Unit Test](./docs/testing/gtest.md)

If you are verifying correct behavior (especially interop problems) **DO NOT**

write a unit test as this not a good use of your time. Write a [cert test](./cert_test.md) instead

so it applies to any stack.

*   [GD Certification Tests](./docs/testing/cert_test.md)

# Gabeldorsche Architecture

[TOC]

This document outlines some architectural considerations we've made when

developing the Gabeldorsche (GD) Bluetooth stack.

## Threading model

First of all, the GD stack does not build on concepts of threads. Instead, it

works with [`Handlers`](#handler). However, since GD ultimately runs on an OS,

it still needs to interact with processes and threads before achieving the

[`Handler`](#handler) abstraction.

### Processes

In general. three types of processes exist in the GD runtime environment:

**Application processes**

:   include third-party apps, other system components such as audio and telecom

    services that interact with the _Bluetooth stack process_ APIs defined

    through various RPC/IPC methods such as Binder, Socket IPC, gRPC, DBUS. and

    so on, using languages such as AIDL or Protobuf. For Android applications,

    although APIs are defined in AIDL, some boiler plate code is wrapped in Java

    libraries exposed through code in

    [`frameworks/base/core/java/android/bluetooth`](https://android.googlesource.com/platform/frameworks/base/+/refs/heads/master/core/java/android/bluetooth/)

    that is released to developers as

    [Android SDK](https://developer.android.com/guide/topics/connectivity/bluetooth).

**Hardware abstraction layer (HAL) processes**

:   one or many processes from the vendor partition, and hence is hardware

    depenedent. They interact with the _Bluetooth stack process_ via a set of

    hardware abstraction APIs defined through RPC/IPC methods such as Binder,

    Socket IPC, DBUS, and so on, using languages such as HIDL. On Android, this

    would be HAL processes that implement HIDL APIs such as

    [IBluetoothHci](https://android.googlesource.com/platform/hardware/interfaces/+/refs/heads/master/bluetooth/1.1/IBluetoothHci.hal)

    and

    [IBluetoothAudioProvider](https://android.googlesource.com/platform/hardware/interfaces/+/refs/heads/master/bluetooth/audio/2.0/IBluetoothAudioProvider.hal).

**Bluetooth stack process**

:   typically one single process that implements various Bluetooth protocols and

    profiles above the Host Controller Interface (HCI) and below the Bluetooth

    SDK APIs. On one hand, it servces the requests from _Application processes_;

    on the other hand, it forwards these requests via interactions with _HAL

    processes_. On Android, this process typically runs under AID_BLUETOOTH

    (usually 1002) with process name "com.android.bluetooth". The process is

    started in Java and loads native libraries through JNI. Other systems that

    do not use Java virtual machine may have a pure native process. Multiple

    threads may exist in this process for various reasons. The GD stack runs

    entirely in this process.

### Threads in Bluetooth stack process

Currently, the goals of thread optimization in the Bluetooth stack are:

*   Reduce the number of threads as much as possible to simplify synchronization

*   Do blocking I/O operations in separate threads

*   Try moving I/O operations into polling mode so that we can use event driven

    methods to interact with it on main thread

*   Move alarm and timer mechanisms to their calling threads to avoid a separate

    alarm thread

*   Isolate individual components so that each component can be started and

    stopped individually without terminating the main thread

*   Prefer data passing over data sharing among threads to reduce locking and

    race conditions

After above optimization, we are left with five main types of threads within the

native code:

**Main thread**

:   The main workhorse in the Bluetooth stack. The thread's execution context is

    further divided into [`Handlers`](#handler) that reside in individual

    [`Modules`](#module). This thread can be divided further into smaller ones

    if performance is constrained on the running platform. Deployer just needs

    to bind handlers to different threads and that should not affect the overall

    operation.

**JNI thread**

:   In the native thread, we treat the Java layer as a separate application as

    its threading module is completely different. Therefore, we put a thread

    between these two layers to buffer any blocking operation.

**HCI thread (or other HW I/O thread)**

:   This thread is responsible for deadling with hardware I/O and can be

    potentially blocking. Hence it has its separate thread to avoid blocking the

    main thread.

**Audio worker thread**

:   Responsible for audio encoding and decoding operations that require higher

    precision execution timing. Such worker has its separate thread to avoid

    being affected by the main thread.

**Socket I/O thread**

:   Communicate with various applications that uses the

    [`BluetootSocket`](https://developer.android.com/reference/android/bluetooth/BluetoothSocket)

    interface. It has its sepearate thread due to potential I/O delay.

### Data flow diagram

Function invocations between different components are abstracted as control

packets (function closure) passed through queues. Data flow between components

are data packets sent through queues, signaled using [`Reactor`](#reactor). They

will merge to the input queue for each component. We define three types of

queues:

**Non-blocking queue**

:   When users try to dequeue when it’s empty, or enqueue when it’s full, it

    will return immediately. All queueing within a thread must be non-blocking,

    because otherwise it will deadlock.

**Blocking queue**

:   When users try to dequeue when it’s empty, or enqueue when it’s full, it

    will block, until other thread makes the queue to be writable/readable. It

    can be used as a flow control mechanism to avoid too many packets from user

    thread.

**Leaky queue**

:   Same as non-blocking queue, but it will flush when it’s full and user tries

    to enqueue. This is useful for audio encoding.

![Threading Model](./data_flow_diagram.svg)

## Building blocks

### Module {#module}

Code in GD is packed into C++ objects called

[`Module`](https://android.googlesource.com/platform/packages/modules/Bluetooth/system/+/master/gd/module.h).

A module standardized the following aspects of GD code:

*   **Dependencies**: A module provides its own dependencies on other modules by

    implementing `ListDependencies()`

*   **Life Cycle**: A module must implement `Start()` and `Stop()` life cycle

    methods

*   **Threading Module**: The `Module` base class provides a `Handler` for code

    execution context via `GetHandler()`

*   **Metrics**: A `Module` can dump its state information for dumpsys through

    `DumpState()`

See its definition at: https://android.googlesource.com/platform/packages/modules/Bluetooth/system/+/master/gd/module.h

### Handler {#handler}

Similar to

[`android.os.Handler`](https://developer.android.com/reference/android/os/Handler),

[`bluetooth::os::Handler`](https://android.googlesource.com/platform/packages/modules/Bluetooth/system/+/master/gd/os/handler.h)

provides a sequential execution context while hiding the concept of thread from

the executing code.

By scoping execution context into smaller areas, `Handler` benefits development

in the following ways:

*   Less need for locking due to sequential execution context

*   Smaller context leads to easier management of code flow

*   Separation from thread gives system deployer more freedom to tweak the

    underlying thread allocation. For example, for real time OS without full

    thread implementation, a `Handler` can be used to provide a near-thread

    execution context

Of course, there are downsides of using `Handler`, which developers should be

cautious about:

WARNING: Although multiple `Handler` could bind to the same thread, `Handler`

does not gurantee sequential execution of code across different `Handler` even

when the are on the same thread.

WARNING: Locking among `Handlers` that were bound to the same thread may result

in deadlock

WARNING: Data must be copied between `Handler` to avoid both deadlock and race

condition

See its definition at: https://android.googlesource.com/platform/packages/modules/Bluetooth/system/+/master/gd/os/handler.h

### Reactor {#reactor}

[`bluetooth::os:Reactor`](https://android.googlesource.com/platform/packages/modules/Bluetooth/system/+/master/gd/os/reactor.h)

implements the

[Reactor Design Pattern](https://en.wikipedia.org/wiki/Reactor_pattern), in

which concurrent _Events_ are demultiplexed by a _Synchronous Event

Demultiplexer_ to a list of _Request Handlers_ registered through a

_Dispatcher_.

In a generic Linux operating system, such as Android, we implemented it using

file descriptors such as

[eventfd](http://man7.org/linux/man-pages/man2/eventfd.2.html) for `Handler`,

[timerfd](http://man7.org/linux/man-pages/man2/timerfd_create.2.html) for

`Alarm`, and [socketfd](http://man7.org/linux/man-pages/man2/socket.2.html) for

data processing pipelines. In the context of file descriptors, events are

catigorized into two types:

*   **OnReadReady**: means that the demultiplexer has some events for the

    handler and the handler can read at least one event from the underlying

    event queue. This is often associated with `EPOLLIN`, `EPOLLHUP`,

    `EPOLLRDHUP`, and `EPOLLERR`.

*   **OnWriteReady**: means that the demultiplexer is ready to consume more

    events from this handler, and the handler can write at least one event to

    the underlying queue. this is often associated with `EPOLLOUT`.

This pattern naturally creates a back pressure from one queue to another without

any extra signaling mechanism. When used in networking stack like ours, it

simplifies the signaling code flow.

See its definition at:

https://android.googlesource.com/platform/packages/modules/Bluetooth/system/+/master/gd/os/reactor.h

A pure data use case of `Reactor` is a `Reactive Queue`, see its definition at:

https://android.googlesource.com/platform/packages/modules/Bluetooth/system/+/master/gd/os/queue.h

## Packet Definition Language (PDL)

Packet parsing and serialization has been a big part of any networking stack. It

is usually the first snippet of code that interface with a remote device. In the

past, this has been achieved manually using macros like `STREAM_TO_UNIT8` or

`UINT8_TO_STREAM`. This manual method is tedious and errorprone. To fix this, we

created a Packet Definition Language that defines networking packet structure to

the bits level. C++ headers and Python bindings will be automatically generated

from its code generator and any fixes to the code generator will apply

systematically to all packet code generated.

Example PDL:

```

// Comments

little_endian_packets // Whether this packet is big or small endian

// Include header from other C++ header files

custom_field SixBytes : 48 "packet/parser/test/" // expect six_bytes.h

custom_field Variable "packet/parser/test/" // expect variable.h

// A packet

packet Parent {

  _fixed_ = 0x12 : 8, // fixed field 0x12 that takes 8 bits

  _size_(_payload_) : 8, // Size field that takes 8 bits

  _payload_, // special payload field of variable size

  footer : 8, // fiexed size footer of 8 bits

}

packet Child : Parent {

  field_name : 16, // addition field append after Parent

}

// an enum of 4 bits

enum FourBits : 4 {

  ONE = 1,

  TWO = 2,

  THREE = 3,

  FIVE = 5,

  TEN = 10,

  LAZY_ME = 15,

}

```

See its documentation at:

https://android.googlesource.com/platform/packages/modules/Bluetooth/system/+/master/gd/packet/parser/README

## Calling convention between modules

### Asynchronous server-client model

For most communication among modules, developers should assume an asynchronous

server-client model in a generic model like:

```c++

// Define callback function type

using CallbackFunction = std::function<void(ParamType)>;

// Asynchronous method definition

bool Foo(Parameter param, CallbackFunction callback);

// A new callback is passed for each asynchronous call

// Always prefer lambda over std::bind

CallbackFunction callback = [this] {

  // do something

};

Parameter param = {

  // something

};

if (Foo(param, callback)) {

   // The callback will be invoked

   // Callback must be invoked in the future

} else {

   // Failed, no need to wait

}

```

Many protocols and profiles fit into such model such as `AclManager` and

`L2cap`.

### Synchronous database model

In some cases, an asynchronous server-client model is not feasible. In this

case, developers can consider a synchronous database model. In such a model,

operations can happen synchronously with help of mutex. When the method returns,

the changes must be reflected to all dependencies. Any changes in the internal

states must be applied atomically.

```c++

// Synchronous method definition

void Foo(Parameter param, Output* output);

int Bar(Parameter param);

Parameter param = {

  // something

};

Output output = {};

Foo(param, &output);

// output can be used immediately

int bar_output = Bar(param);

// bar_output can be used immediately

```

Many storage and informational modules fit into this model such as `Metrics` and

`Storage`.

# Gabeldorsche Style Guide

[TOC]

## Base

In general, when not mentioned in this document, developers should follow the

Google C++ and Google Java style guide as much as possible.

### Google C++ Style Guide

C++ Style Guide: https://google.github.io/styleguide/cppguide.html

### Android and Google Java Style Guide

1.  Android Java Style Guide:

    https://source.android.com/setup/contribute/code-style

2.  when not covered by (1), see External Java Style Guide:

    https://google.github.io/styleguide/javaguide.html

line length limit is 120 characters for C++ and Java

### Python Style Guide

The GD stack uses the Google Python Style Guide:

*   http://google.github.io/styleguide/pyguide.html

with the following modifications as shown in the

[.style.yapf](https://android.googlesource.com/platform/packages/modules/Bluetooth/system/+/refs/heads/master/.style.yapf) definition:

```yapf

based_on_style: google

indent_width: 4

```

## Build files

*   One build target for the entire stack in packages/modules/Bluetooth/system (i.e. one cc_library())

    *   If only part of the stack needs to be compiled, configure it using the

        “target” configuration in Android.bp

*   One build target for all unit tests (i.e. one cc_test)

*   When needed, filgroup() can be created in Android.bp in sub-directories. The

    main build target should use these filegroups() to build the main output

    library.

*   All targets must have host_supported == true unless it is dependent on the

    OS

*   If the stack needs to be compiled using other build system, then the build

    files should also live in packages/modules/Bluetooth/system

## Namespace and include

*   Namespace must follow directory names

*   Top level namespace for internal code is “bluetooth”

*   Top level namespace for externally visible code is “android::bluetooth”

*   Include path must be relative to the root directory of the stack. Normally

    it is packages/modules/Bluetooth/system, for GD refactor code, it is packages/modules/Bluetooth/system/gd

## Multiple implementations of the same header

Since GD interact with many lower level components that are platform dependent,

frequently there is need to implement the same header multiple times for

different platform or hardware. When doing this:

*   Avoid #define macros as much as possible. Instead put code into different

    source files and selectively compile them for different targets.

*   Convention of operating system used:

    *   android/

        *   All Android devices that use HIDL

    *   linux/

        *   All non-Android linux devices

    *   linux_generic/

        *   Android and non-Android linux devices

## Directory structure

Root directory under Android tree:

[**packages/modules/Bluetooth/system/gd/**](https://android.googlesource.com/platform/packages/modules/Bluetooth/system/+/refs/heads/master/gd/)

*   Directory structure should be as flat as possible

*   Each file should contain at most one class

*   Header, source code, and unit test should live in the same directory with

    the following naming guideline:

    *   Source: bb.cc

    *   Header: bb.h

    *   Test: bb_test.cc

*   Each profile should have its own directory and module

*   Source and sink, server and client profiles should live in two sub folders

    of the same common directory where common code can be stored. However,

    source and sink must have separate modules

*   Module file is also the external API header

*   Prefer underscore over dashes

### Example: utility library with OS dependent implementation

*   os/: OS dependent classes such as Alarm, Thread, Handler

    *   Android.bp: Build file that defines file groups that would include

        different source files based on compile time target

    *   alarm.h: common header for alarm

    *   linux_generic/: Implementations for generic Linux OS

        *   alarm.cc: Linux generic implementation of alarm.h using timer_fd

        *   alarm_test.cc: unit test for alarm.h

    *   fuzz/: library needed for fuzz tests in the os/ library

### Example: module with hardware dependent implementation

*   hal/: Hardware abstraction layer such as HCI interfaces, Audio interfaces

    *   Android.bp: Build file that defines file groups that would include

        different source files based on compile time target

    *   hci_hal.h: common library header

    *   hci_hal_android_hidl.cc: implementation of hci_hal.h using Android HIDL

    *   hci_hal_android_hidl_test.cc: unit tests for the Android HIDL

        implementation

    *   hci_hal_host_rootcanal.cc: implementation of hci_hal.h using root-canal

        emulator

    *   hci_hal_host_rootcanal_test.cc: unit tests for the root-canal emulator

        implementation

    *   facade.proto: gRPC automation interface definition for this layer

    *   facade.h/cc: an implementation of the above gRPC interface for the GD

        stack

    *   cert/: certification tests for this module

    *   fuzz/: library needed for fuzz tests in the hal/ module

### Example: similar protocol with the same base

*   l2cap/: L2CAP layer, splitted among classic and LE

    *   classic/: Classic L2CAP module

        *   cert/: certification tests for this module

        *   internal/: internal code to be used only in classic

        *   # Source code and headers being exported to other modules

    *   le/: LE L2CAP module

        *   cert/: certification tests for this module

        *   internal/: internal code to be used only in classic

        *   # Source code and headers being exported to other modules

    *   internal/: L2CAP internal code that should not be used by sources

        outside L2CAP

        *   data_pipeline_manager.h

        *   data_pipeline_manager.cc

        *   data_pipeline_manager_mock.h: Mock of this class, used in unit tests

        *   dynamic_channel_allocator.h

        *   dynamic_channel_allocator.cc

        *   dynamic_channel_allocator_test.cc: GTest unit test of this class

        *   dynamic_channel_allocator_fuzz_test.cc: Fuzz test of this class

    *   *.h/.cc: Common headers and sources that is exposed to other modules

### Example: protocol or profiles with client and server side implementations

*   a2dp/: A2DP profile

    *   sink/: A2DP sink module (e.g. headset)

    *   source/: A2DP source module (e.g. phone)

*   avrcp/

    *   controller/: AVRCP peripheral module (e.g. carkit)

    *   target/: AVRCP target module (e.g. Phone)

*   hfp/

    *   hf/: Handsfree device (e.g. headset)

    *   ag/: Audio gateway (e.g. phone)

## External libraries

To maintain high portability, we are trying to stick with C++ STL as much as

possible. Hence, before including an external library, please ask the team for

review.

Examples of currently used libraries:

*   boringssl: Google's openssl implementation

*   small parts of libchrome, to be removed or replaced eventually

    *   base::OnceCallback

    *   base::Callback

    *   base::BindOnce

    *   base::Bind

*   google-breakpad: host binary crash handler

*   libbacktrace: print stacktrace on crash on host

## Exposed symbols

Given that entire Fluoride library is held in libbluetooth.so dynamic library

file, we need a way to load this library and extract entry points to it. The

only symbols that should be exposed are:

*   An entry point to a normal running adapter module libbluetooth.so

*   A header library to all exposed API service to profiles and layers

*   An entry point to a certification interface, libbluetooth\_certification.so

*   A header library to this certification stack

## Logging

Gabeldorsche uses `printf` style logging with macros defined in `os/log.h`. Five

log levels are available.

*   LOG_VERBOSE(fmt, args...): Will be disabled by default

*   LOG_DEBUG(fmt, args...): Will be disabled by default

*   LOG_INFO(fmt, args...): Enabled

*   LOG_WARN(fmt, args...): Enabled

*   LOG_ERROR(fmt, args...): Enabled

*   LOG_ALWAYS_FATAL(fmt, args...): Enabled, will always crash

*   ASSERT(condition): Enabled, will crash when condition is false

*   ASSERT_LOG(conditon, fmt, args...): Enabled, will crash and print log when

    condition is false

In general, errors that are caused by remote device should never crash our stack

and should be logged using LOG_WARN() only. Recoverable errors due to our stack

or badly behaved bluetooth controller firmware should be logged using

LOG_ERROR() before recovery. Non-recoverable errors should be logged as

LOG_ALWAYS_FATAL() to crash the stack and restart.