misc: mic: SCIF header file and IOCTL interface (7df20f2d) · Commits · e / devices / android_kernel_fairphone_FP4

Documentation/mic/mic_overview.txt

+16 −12

Original line number	Diff line number	Diff line
		@@ -24,6 +24,10 @@ a virtual bus called mic bus is created and virtual dma devices are
		created on it by the host/card drivers. On host the channels are private
		and used only by the host driver to transfer data for the virtio devices.

		The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a
		low level communications API across PCIe currently implemented for MIC.
		More details are available at scif_overview.txt.

		Here is a block diagram of the various components described above. The
		virtio backends are situated on the host rather than the card given better
		single threaded performance for the host compared to MIC, the ability of
		@@ -47,18 +51,18 @@ the fact that the virtio block storage backend can only be on the host.
		\| \| \| Virtio over PCIe IOCTLs \|
		\| \| +--------------------------+
		+-----------+ \| \| \| +-----------+
		\| MIC DMA \| \| \| \| \| MIC DMA \|
		\| Driver \| \| \| \| \| Driver \|
		+-----------+ \| \| \| +-----------+
		\| \| \| \| \|
		+---------------+ \| \| \| +----------------+
		\|MIC virtual Bus\| \| \| \| \|MIC virtual Bus \|
		+---------------+ \| \| \| +----------------+
		\| \| \| \| \|
		\| +--------------+ \| +---------------+ \|
		\| \|Intel MIC \| \| \|Intel MIC \| \|
		+---\|Card Driver \| \| \|Host Driver \| \|
		+--------------+ \| +---------------+-----+
		\| MIC DMA \| \| +----------+ \| +-----------+ \| \| MIC DMA \|
		\| Driver \| \| \| SCIF \| \| \| SCIF \| \| \| Driver \|
		+-----------+ \| +----------+ \| +-----------+ \| +-----------+
		\| \| \| \| \| \| \|
		+---------------+ \| +-----+-----+ \| +-----+-----+ \| +---------------+
		\|MIC virtual Bus\| \| \|SCIF HW Bus\| \| \|SCIF HW BUS\| \| \|MIC virtual Bus\|
		+---------------+ \| +-----------+ \| +-----+-----+ \| +---------------+
		\| \| \| \| \| \| \|
		\| +--------------+ \| \| \| +---------------+ \|
		\| \|Intel MIC \| \| \| \| \|Intel MIC \| \|
		+---\|Card Driver +----+ \| \| \|Host Driver \| \|
		+--------------+ \| +----+---------------+-----+
		\| \| \|
		+-------------------------------------------------------------+
		\| \|

Documentation/mic/scif_overview.txt

0 → 100644

+98 −0

Original line number	Diff line number	Diff line
		The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low
		level communications API across PCIe currently implemented for MIC. Currently
		SCIF provides inter-node communication within a single host platform, where a
		node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of
		communicating over the PCIe bus while providing an API that is symmetric
		across all the nodes in the PCIe network. An important design objective for SCIF
		is to deliver the maximum possible performance given the communication
		abilities of the hardware. SCIF has been used to implement an offload compiler
		runtime and OFED support for MPI implementations for MIC coprocessors.

		==== SCIF API Components ====
		The SCIF API has the following parts:
		1. Connection establishment using a client server model
		2. Byte stream messaging intended for short messages
		3. Node enumeration to determine online nodes
		4. Poll semantics for detection of incoming connections and messages
		5. Memory registration to pin down pages
		6. Remote memory mapping for low latency CPU accesses via mmap
		7. Remote DMA (RDMA) for high bandwidth DMA transfers
		8. Fence APIs for RDMA synchronization

		SCIF exposes the notion of a connection which can be used by peer processes on
		nodes in a SCIF PCIe "network" to share memory "windows" and to communicate. A
		process in a SCIF node initiates a SCIF connection to a peer process on a
		different node via a SCIF "endpoint". SCIF endpoints support messaging APIs
		which are similar to connection oriented socket APIs. Connected SCIF endpoints
		can also register local memory which is followed by data transfer using either
		DMA, CPU copies or remote memory mapping via mmap. SCIF supports both user and
		kernel mode clients which are functionally equivalent.

		==== SCIF Performance for MIC ====
		DMA bandwidth comparison between the TCP (over ethernet over PCIe) stack versus
		SCIF shows the performance advantages of SCIF for HPC applications and runtimes.

		Comparison of TCP and SCIF based BW

		Throughput (GB/sec)
		8 + PCIe Bandwidth ******
		+ TCP ######
		7 + ************************************** SCIF %%%%%%
		\| %%%%%%%%%%%%%%%%%%%
		6 + %%%%
		\| %%
		\| %%%
		5 + %%
		\| %%
		4 + %%
		\| %%
		3 + %%
		\| %
		2 + %%
		\| %%
		\| %
		1 +
		+ ######################################
		0 +++---+++--+--+-+--+--+-++-+--+-++-+--+-++-+-
		1 10 100 1000 10000 100000
		Transfer Size (KBytes)

		SCIF allows memory sharing via mmap(..) between processes on different PCIe
		nodes and thus provides bare-metal PCIe latency. The round trip SCIF mmap
		latency from the host to an x100 MIC for an 8 byte message is 0.44 usecs.

		SCIF has a user space library which is a thin IOCTL wrapper providing a user
		space API similar to the kernel API in scif.h. The SCIF user space library
		is distributed @ https://software.intel.com/en-us/mic-developer

		Here is some pseudo code for an example of how two applications on two PCIe
		nodes would typically use the SCIF API:

		Process A (on node A) Process B (on node B)

		/* get online node information */
		scif_get_node_ids(..) scif_get_node_ids(..)
		scif_open(..) scif_open(..)
		scif_bind(..) scif_bind(..)
		scif_listen(..)
		scif_accept(..) scif_connect(..)
		/* SCIF connection established */

		/* Send and receive short messages */
		scif_send(..)/scif_recv(..) scif_send(..)/scif_recv(..)

		/* Register memory */
		scif_register(..) scif_register(..)

		/* RDMA */
		scif_readfrom(..)/scif_writeto(..) scif_readfrom(..)/scif_writeto(..)

		/* Fence DMAs */
		scif_fence_signal(..) scif_fence_signal(..)

		mmap(..) mmap(..)

		/* Access remote registered memory */

		/* Close the endpoints */
		scif_close(..) scif_close(..)

include/linux/scif.h

0 → 100644

+993 −0

File added.

Preview size limit exceeded, changes collapsed.

include/uapi/linux/Kbuild

+1 −0

Original line number	Diff line number	Diff line
		@@ -352,6 +352,7 @@ header-y += rtc.h
		header-y += rtnetlink.h
		header-y += scc.h
		header-y += sched.h
		header-y += scif_ioctl.h
		header-y += screen_info.h
		header-y += sctp.h
		header-y += sdla.h

include/uapi/linux/scif_ioctl.h

0 → 100644

+130 −0

Original line number	Diff line number	Diff line
		/*
		* Intel MIC Platform Software Stack (MPSS)
		*
		* This file is provided under a dual BSD/GPLv2 license. When using or
		* redistributing this file, you may do so under either license.
		*
		* GPL LICENSE SUMMARY
		*
		* Copyright(c) 2014 Intel Corporation.
		*
		* This program is free software; you can redistribute it and/or modify
		* it under the terms of version 2 of the GNU General Public License as
		* published by the Free Software Foundation.
		*
		* This program is distributed in the hope that it will be useful, but
		* WITHOUT ANY WARRANTY; without even the implied warranty of
		* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
		* General Public License for more details.
		*
		* BSD LICENSE
		*
		* Copyright(c) 2014 Intel Corporation.
		*
		* Redistribution and use in source and binary forms, with or without
		* modification, are permitted provided that the following conditions
		* are met:
		*
		* * Redistributions of source code must retain the above copyright
		* notice, this list of conditions and the following disclaimer.
		* * Redistributions in binary form must reproduce the above copyright
		* notice, this list of conditions and the following disclaimer in
		* the documentation and/or other materials provided with the
		* distribution.
		* * Neither the name of Intel Corporation nor the names of its
		* contributors may be used to endorse or promote products derived
		* from this software without specific prior written permission.
		*
		* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
		* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
		* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
		* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
		* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
		* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
		* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
		* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
		* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
		* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
		* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
		*
		* Intel SCIF driver.
		*
		*/
		/*
		* -----------------------------------------
		* SCIF IOCTL interface information
		* -----------------------------------------
		*/
		#ifndef SCIF_IOCTL_H
		#define SCIF_IOCTL_H

		#include <linux/types.h>

		/**
		* struct scif_port_id - SCIF port information
		* @node: node on which port resides
		* @port: local port number
		*/
		struct scif_port_id {
		__u16 node;
		__u16 port;
		};

		/**
		* struct scifioctl_connect - used for SCIF_CONNECT IOCTL
		* @self: used to read back the assigned port_id
		* @peer: destination node and port to connect to
		*/
		struct scifioctl_connect {
		struct scif_port_id self;
		struct scif_port_id peer;
		};

		/**
		* struct scifioctl_accept - used for SCIF_ACCEPTREQ IOCTL
		* @flags: flags
		* @peer: global id of peer endpoint
		* @endpt: new connected endpoint descriptor
		*/
		struct scifioctl_accept {
		__s32 flags;
		struct scif_port_id peer;
		__u64 endpt;
		};

		/**
		* struct scifioctl_msg - used for SCIF_SEND/SCIF_RECV IOCTL
		* @msg: message buffer address
		* @len: message length
		* @flags: flags
		* @out_len: number of bytes sent/received
		*/
		struct scifioctl_msg {
		__u64 msg;
		__s32 len;
		__s32 flags;
		__s32 out_len;
		};

		/**
		* struct scifioctl_node_ids - used for SCIF_GET_NODEIDS IOCTL
		* @nodes: pointer to an array of node_ids
		* @self: ID of the current node
		* @len: length of array
		*/
		struct scifioctl_node_ids {
		__u64 nodes;
		__u64 self;
		__s32 len;
		};

		#define SCIF_BIND _IOWR('s', 1, __u64)
		#define SCIF_LISTEN _IOW('s', 2, __s32)
		#define SCIF_CONNECT _IOWR('s', 3, struct scifioctl_connect)
		#define SCIF_ACCEPTREQ _IOWR('s', 4, struct scifioctl_accept)
		#define SCIF_ACCEPTREG _IOWR('s', 5, __u64)
		#define SCIF_SEND _IOWR('s', 6, struct scifioctl_msg)
		#define SCIF_RECV _IOWR('s', 7, struct scifioctl_msg)
		#define SCIF_GET_NODEIDS _IOWR('s', 14, struct scifioctl_node_ids)

		#endif /* SCIF_IOCTL_H */

Original line number	Diff line number	Diff line
		@@ -24,6 +24,10 @@ a virtual bus called mic bus is created and virtual dma devices are
		created on it by the host/card drivers. On host the channels are private
		and used only by the host driver to transfer data for the virtio devices.

		The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a
		low level communications API across PCIe currently implemented for MIC.
		More details are available at scif_overview.txt.

		Here is a block diagram of the various components described above. The
		virtio backends are situated on the host rather than the card given better
		single threaded performance for the host compared to MIC, the ability of
		@@ -47,18 +51,18 @@ the fact that the virtio block storage backend can only be on the host.
		\| \| \| Virtio over PCIe IOCTLs \|
		\| \| +--------------------------+
		+-----------+ \| \| \| +-----------+
		\| MIC DMA \| \| \| \| \| MIC DMA \|
		\| Driver \| \| \| \| \| Driver \|
		+-----------+ \| \| \| +-----------+
		\| \| \| \| \|
		+---------------+ \| \| \| +----------------+
		\|MIC virtual Bus\| \| \| \| \|MIC virtual Bus \|
		+---------------+ \| \| \| +----------------+
		\| \| \| \| \|
		\| +--------------+ \| +---------------+ \|
		\| \|Intel MIC \| \| \|Intel MIC \| \|
		+---\|Card Driver \| \| \|Host Driver \| \|
		+--------------+ \| +---------------+-----+
		\| MIC DMA \| \| +----------+ \| +-----------+ \| \| MIC DMA \|
		\| Driver \| \| \| SCIF \| \| \| SCIF \| \| \| Driver \|
		+-----------+ \| +----------+ \| +-----------+ \| +-----------+
		\| \| \| \| \| \| \|
		+---------------+ \| +-----+-----+ \| +-----+-----+ \| +---------------+
		\|MIC virtual Bus\| \| \|SCIF HW Bus\| \| \|SCIF HW BUS\| \| \|MIC virtual Bus\|
		+---------------+ \| +-----------+ \| +-----+-----+ \| +---------------+
		\| \| \| \| \| \| \|
		\| +--------------+ \| \| \| +---------------+ \|
		\| \|Intel MIC \| \| \| \| \|Intel MIC \| \|
		+---\|Card Driver +----+ \| \| \|Host Driver \| \|
		+--------------+ \| +----+---------------+-----+
		\| \| \|
		+-------------------------------------------------------------+
		\| \|