msm: rdbg: Add snapshot of remote debugger driver

Introduction

============

The goal of this debug feature is to provide a reliable, responsive,

accurate and secure debug capability to developers interested in

debugging MSM subsystem processor images without the use of a hardware

debugger.

The Debug Agent along with the Remote Debug Driver implements a shared

memory based transport mechanism that allows for a debugger (ex. GDB)

running on a host PC to communicate with a remote stub running on

peripheral subsystems such as the ADSP, MODEM etc.

The diagram below depicts end to end the components involved to

support remote debugging:

:               :

:    HOST (PC)  :  MSM

:  ,--------,   :   ,-------,

:  |        |   :   | Debug |                         ,--------,

:  |Debugger|<--:-->| Agent |                         | Remote |

:  |        |   :   |  App  |                  +----->| Debug  |

:  `--------`   :   |-------|    ,--------,    |      | Stub   |

:               :   | Remote|    |        |<---+      `--------`

:               :   | Debug |<-->|--------|

:               :   | Driver|    |        |<---+      ,--------,

:               :   `-------`    `--------`    |      | Remote |

:               :       LA         Shared      +----->| Debug  |

:               :                  Memory             | Stub   |

:               :                                     `--------`

:               :                               Peripheral Subsystems

:               :                                 (ADSP, MODEM, ...)

Debugger:       Debugger application running on the host PC that

                communicates with the remote stub.

                Examples: GDB, LLDB

Debug Agent:    Software that runs on the Linux Android platform

                that provides connectivity from the MSM to the

                host PC. This involves two portions:

                1) User mode Debug Agent application that discovers

                processes running on the subsystems and creates

                TCP/IP sockets for the host to connect to. In addition

                to this, it creates an info (or meta) port that

                users can connect to discover the various

                processes and their corresponding debug ports.

Remote Debug    A character based driver that the Debug

Driver:         Agent uses to transport the payload received from the

                host to the debug stub running on the subsystem

                processor over shared memory and vice versa.

Shared Memory:  Shared memory from the SMEM pool that is accessible

                from the Applications Processor (AP) and the

                subsystem processors.

Remote Debug    Privileged code that runs in the kernels of the

Stub:           subsystem processors that receives debug commands

                from the debugger running on the host and

                acts on these commands. These commands include reading

                and writing to registers and memory belonging to the

                subsystem's address space, setting breakpoints,

                single stepping etc.

Hardware description

====================

The Remote Debug Driver interfaces with the Remote Debug stubs

running on the subsystem processors and does not drive or

manage any hardware resources.

Software description

====================

The debugger and the remote stubs use Remote Serial Protocol (RSP)

to communicate with each other. This is widely used protocol by both

software and hardware debuggers. RSP is an ASCII based protocol

and used when it is not possible to run GDB server on the target under

debug.

The Debug Agent application along with the Remote Debug Driver

is responsible for establishing a bi-directional connection from

the debugger application running on the host to the remote debug

stub running on a subsystem. The Debug Agent establishes connectivity

to the host PC via TCP/IP sockets.

This feature uses ADB port forwarding to establish connectivity

between the debugger running on the host and the target under debug.

Please note the Debug Agent does not expose HLOS memory to the

remote subsystem processors.

Design

======

Here is the overall flow:

1) When the Debug Agent application starts up, it opens up a shared memory

based transport channel to the various subsystem processor images.

2) The Debug Agent application sends messages across to the remote stubs

to discover the various processes that are running on the subsystem and

creates debug sockets for each of them.

3) Whenever a process running on a subsystem exits, the Debug Agent

is notified by the stub so that the debug port and other resources

can be reclaimed.

4) The Debug Agent uses the services of the Remote Debug Driver to

transport payload from the host debugger to the remote stub and vice versa.

5) Communication between the Remote Debug Driver and the Remote Debug stub

running on the subsystem processor is done over shared memory (see figure).

SMEM services are used to allocate the shared memory that will

be readable and writeable by the AP and the subsystem image under debug.

A separate SMEM allocation takes place for each subsystem processor

involved in remote debugging. The remote stub running on each of the

subsystems allocates a SMEM buffer using a unique identifier so that both

the AP and subsystem get the same physical block of memory. It should be

noted that subsystem images can be restarted at any time.

However, when a subsystem comes back up, its stub uses the same unique

SMEM identifier to allocate the SMEM block. This would not result in a

new allocation rather the same block of memory in the first bootup instance

is provided back to the stub running on the subsystem.

An 8KB chunk of shared memory is allocated and used for communication

per subsystem. For multi-process capable subsystems, 16KB chunk of shared

memory is allocated to allow for simultaneous debugging of more than one

process running on a single subsystem.

The shared memory is used as a circular ring buffer in each direction.

Thus we have a bi-directional shared memory channel between the AP

and a subsystem. We call this SMQ. Each memory channel contains a header,

data and a control mechanism that is used to synchronize read and write

of data between the AP and the remote subsystem.

Overall SMQ memory view:

:

:    +------------------------------------------------+

:    | SMEM buffer                                    |

:    |-----------------------+------------------------|

:    |Producer: LA           | Producer: Remote       |

:    |Consumer: Remote       |           subsystem    |

:    |          subsystem    | Consumer: LA           |

:    |                       |                        |

:    |               Producer|                Consumer|

:    +-----------------------+------------------------+

:    |                       |

:    |                       |

:    |                       +--------------------------------------+

:    |                                                              |

:    |                                                              |

:    v                                                              v

:    +--------------------------------------------------------------+

:    |   Header  |       Data      |            Control             |

:    +-----------+---+---+---+-----+----+--+--+-----+---+--+--+-----+

:    |           | b | b | b |     | S  |n |n |     | S |n |n |     |

:    |  Producer | l | l | l |     | M  |o |o |     | M |o |o |     |

:    |    Ver    | o | o | o |     | Q  |d |d |     | Q |d |d |     |

:    |-----------| c | c | c | ... |    |e |e | ... |   |e |e | ... |

:    |           | k | k | k |     | O  |  |  |     | I |  |  |     |

:    |  Consumer |   |   |   |     | u  |0 |1 |     | n |0 |1 |     |

:    |    Ver    | 0 | 1 | 2 |     | t  |  |  |     |   |  |  |     |

:    +-----------+---+---+---+-----+----+--+--+-----+---+--+--+-----+

:                                       |           |

:                                       +           |

:                                                   |

:                          +------------------------+

:                          |

:                          v

:                        +----+----+----+----+

:                        | SMQ Nodes         |

:                        |----|----|----|----|

:                 Node # |  0 |  1 |  2 | ...|

:                        |----|----|----|----|

: Starting Block Index # |  0 |  3 |  8 | ...|

:                        |----|----|----|----|

:            # of blocks |  3 |  5 |  1 | ...|

:                        +----+----+----+----+

:

Header: Contains version numbers for software compatibility to ensure

that both producers and consumers on the AP and subsystems know how to

read from and write to the queue.

Both the producer and consumer versions are 1.

:     +---------+-------------------+

:     | Size    | Field             |

:     +---------+-------------------+

:     | 1 byte  | Producer Version  |

:     +---------+-------------------+

:     | 1 byte  | Consumer Version  |

:     +---------+-------------------+

Data: The data portion contains multiple blocks [0..N] of a fixed size.

The block size SM_BLOCKSIZE is fixed to 128 bytes for header version #1.

Payload sent from the debug agent app is split (if necessary) and placed

in these blocks. The first data block is placed at the next 8 byte aligned

address after the header.

The number of blocks for a given SMEM allocation is derived as follows:

  Number of Blocks = ((Total Size - Alignment - Size of Header

                      - Size of SMQIn - Size of SMQOut)/(SM_BLOCKSIZE))

The producer maintains a private block map of each of these blocks to

determine which of these blocks in the queue is available and which are free.

Control:

The control portion contains a list of nodes [0..N] where N is number

of available data blocks. Each node identifies the data

block indexes that contain a particular debug message to be transferred,

and the number of blocks it took to hold the contents of the message.

Each node has the following structure:

:     +---------+-------------------+

:     | Size    | Field             |

:     +---------+-------------------+

:     | 2 bytes |Staring Block Index|

:     +---------+-------------------+

:     | 2 bytes |Number of Blocks   |

:     +---------+-------------------+

The producer and the consumer update different parts of the control channel

(SMQOut / SMQIn) respectively. Each of these control data structures contains

information about the last node that was written / read, and the actual nodes

that were written/read.

SMQOut Structure (R/W by producer, R by consumer):

:     +---------+-------------------+

:     | Size    | Field             |

:     +---------+-------------------+

:     | 4 bytes | Magic Init Number |

:     +---------+-------------------+

:     | 4 bytes | Reset             |

:     +---------+-------------------+

:     | 4 bytes | Last Sent Index   |

:     +---------+-------------------+

:     | 4 bytes | Index Free Read   |

:     +---------+-------------------+

SMQIn Structure (R/W by consumer, R by producer):

:     +---------+-------------------+

:     | Size    | Field             |

:     +---------+-------------------+

:     | 4 bytes | Magic Init Number |

:     +---------+-------------------+

:     | 4 bytes | Reset ACK         |

:     +---------+-------------------+

:     | 4 bytes | Last Read Index   |

:     +---------+-------------------+

:     | 4 bytes | Index Free Write  |

:     +---------+-------------------+

Magic Init Number:

Both SMQ Out and SMQ In initialize this field with a predefined magic

number so as to make sure that both the consumer and producer blocks

have fully initialized and have valid data in the shared memory control area.

  Producer Magic #: 0xFF00FF01

  Consumer Magic #: 0xFF00FF02

SMQ Out's Last Sent Index and Index Free Read:

  Only a producer can write to these indexes and they are updated whenever

  there is new payload to be inserted into the SMQ in order to be sent to a

  consumer.

  The number of blocks required for the SMQ allocation is determined as:

   (payload size + SM_BLOCKSIZE - 1) / SM_BLOCKSIZE

  The private block map is searched for a large enough continuous set of blocks

  and the user data is copied into the data blocks.

  The starting index of the free block(s) is updated in the SMQOut's Last Sent

  Index. This update keeps track of which index was last written to and the

  producer uses it to determine where the the next allocation could be done.

  Every allocation, a producer updates the Index Free Read from its

  collaborating consumer's Index Free Write field (if they are unequal).

  This index value indicates that the consumer has read all blocks associated

  with allocation on the SMQ and that the producer can reuse these blocks for

  subsquent allocations since this is a circular queue.

  At cold boot and restart, these indexes are initialized to zero and all

  blocks are marked as available for allocation.

SMQ In's Last Read Index and Index Free Write:

  These indexes are written to only by a consumer and are updated whenever

  there is new payload to be read from the SMQ. The Last Read Index keeps

  track of which index was last read by the consumer and using this, it

  determines where the next read should be done.

  After completing a read, Last Read Index is incremented to the

  next block index. A consumer updates Index Free Write to the starting

  index of an allocation whenever it has completed processing the blocks.

  This is an optimization that can be used to prevent an additional copy

  of data from the queue into a client's data buffer and the data in the queue

  itself can be used.

  Once Index Free Write is updated, the collaborating producer (on the next

  data allocation) reads the updated Index Free Write value and it then

  updates its corresponding SMQ Out's Index Free Read and marks the blocks

  associated with that index as available for allocation. At cold boot and

  restart, these indexes are initialized to zero.

SMQ Out Reset# and SMQ In Reset ACK #:

  Since subsystems can restart at anytime, the data blocks and control channel

  can be in an inconsistent state when a producer or consumer comes up.

  We use Reset and Reset ACK to manage this. At cold boot, the producer

  initializes the Reset# to a known number ex. 1. Every other reset that the

  producer undergoes, the Reset#1 is simply incremented by 1. All the producer

  indexes are reset.

  When the producer notifies the consumer of data availability, the consumer

  reads the producers Reset # and copies that into its SMQ In Reset ACK#

  field when they differ. When that occurs, the consumer resets its

  indexes to 0.

6) Asynchronous notifications between a producer and consumer are

done using the SMP2P service which is interrupt based.

Power Management

================

None

SMP/multi-core

==============

The driver uses completion to wake up the Debug Agent client threads.

Security

========

From the perspective of the subsystem, the AP is untrusted. The remote

stubs consult the secure debug fuses to determine whether or not the

remote debugging will be enabled at the subsystem.

If the hardware debug fuses indicate that debugging is disabled, the

remote stubs will not be functional on the subsystem. Writes to the

queue will only be done if the driver sees that the remote stub has been

initialized on the subsystem.

Therefore even if any untrusted software running on the AP requests

the services of the Remote Debug Driver and inject RSP messages

into the shared memory buffer, these RSP messages will be discarded and

an appropriate error code will be sent up to the invoking application.

Performance

===========

During operation, the Remote Debug Driver copies RSP messages

asynchronously sent from the host debugger to the remote stub and vice

versa. The debug messages are ASCII based and relatively short

(<25 bytes) and may once in a while go up to a maximum 700 bytes

depending on the command the user requested. Thus we do not

anticipate any major performance impact. Moreover, in a typical

functional debug scenario performance should not be a concern.

Interface

=========

The Remote Debug Driver is a character based device that manages

a piece of shared memory that is used as a bi-directional

single producer/consumer circular queue using a next fit allocator.

Every subsystem, has its own shared memory buffer that is managed

like a separate device.

The driver distinguishes each subsystem processor's buffer by

registering a node with a different minor number.

For each subsystem that is supported, the driver exposes a user space

interface through the following node:

    - /dev/rdbg-<subsystem>

    Ex. /dev/rdbg-adsp (for the ADSP subsystem)

The standard open(), close(), read() and write() API set is

implemented.

The open() syscall will fail if a subsystem is not present or supported

by the driver or a shared memory buffer cannot be allocated for the

AP - subsystem communication. It will also fail if the subsytem has

not initialized the queue on its side. Here are the error codes returned

in case a call to open() fails:

ENODEV - memory was not yet allocated for the device

EEXIST - device is already opened

ENOMEM - SMEM allocation failed

ECOMM - Subsytem queue is not yet setup

ENOMEM - Failure to initialize SMQ

read() is a blocking call that will return with the number of bytes written

by the subsystem whenever the subsystem sends it some payload. Here are the

error codes returned in case a call to read() fails:

EINVAL - Invalid input

ENODEV - Device has not been opened yet

ERESTARTSYS - call to wait_for_completion_interruptible is interrupted

ENODATA - call to smq_receive failed

write() attempts to send user mode payload out to the subsystem. It can fail

if the SMQ is full. The number of bytes written is returned back to the user.

Here are the error codes returned in case a call to write() fails:

EINVAL - Invalid input

ECOMM - SMQ send failed

In the close() syscall, the control information state of the SMQ is

initialized to zero thereby preventing any further communication between

the AP and the subsystem. Here is the error code returned in case

a call to close() fails:

ENODEV - device wasn't opened/initialized

The Remote Debug driver uses SMP2P for bi-directional AP to subsystem

notification. Notifications are sent to indicate that there are new

debug messages available for processing. Each subsystem that is

supported will need to add a device tree entry per the usage

specification of SMP2P driver.

In case the remote stub becomes non operational or the security configuration

on the subsystem does not permit debugging, any messages put in the SMQ will

not be responded to. It is the responsibility of the Debug Agent app and the

host debugger application such as GDB to timeout and notify the user of the

non availability of remote debugging.

Driver parameters

=================

None

Config options

==============

The driver is configured with a device tree entry to map an SMP2P entry

to the device. The SMP2P entry name used is "rdbg". Please see

kernel\Documentation\arm\msm\msm_smp2p.txt for information about the

device tree entry required to configure SMP2P.

The driver uses the SMEM allocation type SMEM_LC_DEBUGGER to allocate memory

for the queue that is used to share data with the subsystems.

Dependencies

============

The Debug Agent driver requires services of SMEM to

allocate shared memory buffers.

SMP2P is used as a bi-directional notification

mechanism between the AP and a subsystem processor.

User space utilities

====================

This driver is meant to be used in conjunction with the user mode

Remote Debug Agent application.

Other

=====

None

Known issues

============

For targets with an external subsystem, we cannot use

shared memory for communication and would have to use the prevailing

transport mechanisms that exists between the AP and the external subsystem.

This driver cannot be leveraged for such targets.

To do

=====

None

Qualcomm Technologies, Inc. Remote Debugger (RDBG) driver

Required properties:

-compatible : Should be one of

	To communicate with adsp

		qcom,smp2p-interrupt-rdbg-2-in (inbound)

		qcom,smp2p-interrupt-rdbg-2-out (outbound)

	To communicate with cdsp

		qcom,smp2p-interrupt-rdbg-5-in (inbound)

		qcom,smp2p-interrupt-rdbg-5-out (outbound)

Example:

	qcom,smp2p_interrupt_rdbg_2_in {

            compatible = "qcom,smp2p-interrupt-rdbg-2-in";

	};

          applications DSP processor. Say M if you want to enable this

          module.

config MSM_RDBG

	tristate "QTI Remote debug driver"

	help

	  Implements a shared memory based transport mechanism that allows

	  for a debugger running on a host PC to communicate with a remote

	  stub running on peripheral subsystems such as the ADSP, MODEM etc.

endmenu

  obj-$(CONFIG_MSM_ADSPRPC)	+= adsprpc_compat.o

endif

obj-$(CONFIG_MSM_FASTCVPD)	+= fastcvpd.o

obj-$(CONFIG_MSM_RDBG)		+= rdbg.o