Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 7df20f2d authored by Sudeep Dutt's avatar Sudeep Dutt Committed by Greg Kroah-Hartman
Browse files

misc: mic: SCIF header file and IOCTL interface 



This patch introduces the SCIF documentation in the header file
and describes the IOCTL interface for user mode. mic_overview.txt
is updated with documentation on SCIF and a new document
describing SCIF in more details is available in scif_overview.txt.

Reviewed-by: default avatarNikhil Rao <nikhil.rao@intel.com>
Reviewed-by: default avatarAshutosh Dixit <ashutosh.dixit@intel.com>
Signed-off-by: default avatarSudeep Dutt <sudeep.dutt@intel.com>
Signed-off-by: default avatarGreg Kroah-Hartman <gregkh@linuxfoundation.org>
parent 0d09f1a5

Documentation/mic/mic_overview.txt

+16 −12
Original line number Diff line number Diff line
@@ -24,6 +24,10 @@ a virtual bus called mic bus is created and virtual dma devices are 
created on it by the host/card drivers. On host the channels are private

and used only by the host driver to transfer data for the virtio devices.



The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a

low level communications API across PCIe currently implemented for MIC.

More details are available at scif_overview.txt.



Here is a block diagram of the various components described above. The

virtio backends are situated on the host rather than the card given better

single threaded performance for the host compared to MIC, the ability of
@@ -47,18 +51,18 @@ the fact that the virtio block storage backend can only be on the host. 
                      |               |       | Virtio over PCIe IOCTLs  |

                      |               |       +--------------------------+

+-----------+         |               |                   |  +-----------+

| MIC DMA   |         |               |                   |  | MIC DMA   |

| Driver    |         |               |                   |  | Driver    |

+-----------+         |               |                   |  +-----------+

      |               |               |                   |        |

+---------------+     |               |                   |  +----------------+

|MIC virtual Bus|     |               |                   |  |MIC virtual Bus |

+---------------+     |               |                   |  +----------------+

      |               |               |                   |              |

      |   +--------------+            |            +---------------+     |

      |   |Intel MIC     |            |            |Intel MIC      |     |

      +---|Card Driver   |            |            |Host Driver    |     |

          +--------------+            |            +---------------+-----+

| MIC DMA   |         |  +----------+ | +-----------+     |  | MIC DMA   |

| Driver    |         |  |  SCIF    | | |   SCIF    |     |  | Driver    |

+-----------+         |  +----------+ | +-----------+     |  +-----------+

      |               |       |       |       |           |        |

+---------------+     | +-----+-----+ | +-----+-----+     | +---------------+

|MIC virtual Bus|     | |SCIF HW Bus| | |SCIF HW BUS|     | |MIC virtual Bus|

+---------------+     | +-----------+ | +-----+-----+     | +---------------+

      |               |       |       |       |           |              |

      |   +--------------+    |       |       |    +---------------+     |

      |   |Intel MIC     |    |       |       |    |Intel MIC      |     |

      +---|Card Driver   +----+       |       |    |Host Driver    |     |

          +--------------+            |       +----+---------------+-----+

                      |               |                   |

             +-------------------------------------------------------------+

             |                                                             |

Documentation/mic/scif_overview.txt

0 → 100644
+98 −0
Original line number Diff line number Diff line 
The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low

level communications API across PCIe currently implemented for MIC. Currently

SCIF provides inter-node communication within a single host platform, where a

node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of

communicating over the PCIe bus while providing an API that is symmetric

across all the nodes in the PCIe network. An important design objective for SCIF

is to deliver the maximum possible performance given the communication

abilities of the hardware. SCIF has been used to implement an offload compiler

runtime and OFED support for MPI implementations for MIC coprocessors.



==== SCIF API Components ====

The SCIF API has the following parts:

1. Connection establishment using a client server model

2. Byte stream messaging intended for short messages

3. Node enumeration to determine online nodes

4. Poll semantics for detection of incoming connections and messages

5. Memory registration to pin down pages

6. Remote memory mapping for low latency CPU accesses via mmap

7. Remote DMA (RDMA) for high bandwidth DMA transfers

8. Fence APIs for RDMA synchronization



SCIF exposes the notion of a connection which can be used by peer processes on

nodes in a SCIF PCIe "network" to share memory "windows" and to communicate. A

process in a SCIF node initiates a SCIF connection to a peer process on a

different node via a SCIF "endpoint". SCIF endpoints support messaging APIs

which are similar to connection oriented socket APIs. Connected SCIF endpoints

can also register local memory which is followed by data transfer using either

DMA, CPU copies or remote memory mapping via mmap. SCIF supports both user and

kernel mode clients which are functionally equivalent.



==== SCIF Performance for MIC ====

DMA bandwidth comparison between the TCP (over ethernet over PCIe) stack versus

SCIF shows the performance advantages of SCIF for HPC applications and runtimes.



             Comparison of TCP and SCIF based BW



  Throughput (GB/sec)

    8 +                                             PCIe Bandwidth ******

      +                                                        TCP ######

    7 +    **************************************             SCIF %%%%%%

      |                       %%%%%%%%%%%%%%%%%%%

    6 +                   %%%%

      |                 %%

      |               %%%

    5 +              %%

      |            %%

    4 +           %%

      |          %%

    3 +         %%

      |        %

    2 +      %%

      |     %%

      |    %

    1 +

      +    ######################################

    0 +++---+++--+--+-+--+--+-++-+--+-++-+--+-++-+-

      1       10     100      1000   10000   100000

                   Transfer Size (KBytes)



SCIF allows memory sharing via mmap(..) between processes on different PCIe

nodes and thus provides bare-metal PCIe latency. The round trip SCIF mmap

latency from the host to an x100 MIC for an 8 byte message is 0.44 usecs.



SCIF has a user space library which is a thin IOCTL wrapper providing a user

space API similar to the kernel API in scif.h. The SCIF user space library

is distributed @ https://software.intel.com/en-us/mic-developer



Here is some pseudo code for an example of how two applications on two PCIe

nodes would typically use the SCIF API:



Process A (on node A)			Process B (on node B)



/* get online node information */

scif_get_node_ids(..)			scif_get_node_ids(..)

scif_open(..)				scif_open(..)

scif_bind(..)				scif_bind(..)

scif_listen(..)

scif_accept(..)				scif_connect(..)

/* SCIF connection established */



/* Send and receive short messages */

scif_send(..)/scif_recv(..)		scif_send(..)/scif_recv(..)



/* Register memory */

scif_register(..)			scif_register(..)



/* RDMA */

scif_readfrom(..)/scif_writeto(..)	scif_readfrom(..)/scif_writeto(..)



/* Fence DMAs */

scif_fence_signal(..)			scif_fence_signal(..)



mmap(..)				mmap(..)



/* Access remote registered memory */



/* Close the endpoints */

scif_close(..)				scif_close(..)