Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 8700e3e7 authored by Moni Shoua's avatar Moni Shoua Committed by Doug Ledford
Browse files

Soft RoCE driver 

Soft RoCE (RXE) - The software RoCE driver

ib_rxe implements the RDMA transport and registers to the RDMA core
device as a kernel verbs provider. It also implements the packet IO
layer. On the other hand ib_rxe registers to the Linux netdev stack
as a udp encapsulating protocol, in that case RDMA, for sending and
receiving packets over any Ethernet device.  This yields a RDMA
transport over the UDP/Ethernet network layer forming a RoCEv2
compatible device.

The configuration procedure of the Soft RoCE drivers requires
binding to any existing Ethernet network device. This is done with
/sys interface.

A userspace Soft RoCE library (librxe) provides user applications
the ability to run with Soft RoCE devices.  The use of rxe verbs ins
user space requires the inclusion of librxe as a device specifics
plug-in to libibverbs. librxe is packaged separately.

Architecture:

     +-----------------------------------------------------------+
     |                          Application                      |
     +-----------------------------------------------------------+
                            +-----------------------------------+
                            |             libibverbs            |
User                        +-----------------------------------+
                            +----------------+ +----------------+
                            | librxe         | | HW RoCE lib    |
                            +----------------+ +----------------+
+---------------------------------------------------------------+
     +--------------+                           +------------+
     | Sockets      |                           | RDMA ULP   |
     +--------------+                           +------------+
     +--------------+                  +---------------------+
     | TCP/IP       |                  | ib_core             |
     +--------------+                  +---------------------+
                             +------------+ +----------------+
Kernel                       | ib_rxe     | | HW RoCE driver |
                             +------------+ +----------------+
     +------------------------------------+
     | NIC driver                         |
     +------------------------------------+

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     +-----------------------------------------------------------+
     |                          Application                      |
     +-----------------------------------------------------------+
                            +-----------------------------------+
                            |             libibverbs            |
User                        +-----------------------------------+
                            +----------------+ +----------------+
                            | librxe         | | HW RoCE lib    |
                            +----------------+ +----------------+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     +--------------+                           +------------+
     | Sockets      |                           | RDMA ULP   |
     +--------------+                           +------------+
     +--------------+                  +---------------------+
     | TCP/IP       |                  | ib_core             |
     +--------------+                  +---------------------+
                             +------------+ +----------------+
Kernel                       | ib_rxe     | | HW RoCE driver |
                             +------------+ +----------------+
     +------------------------------------+
     | NIC driver                         |
     +------------------------------------+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Soft RoCE resources:

[1[ https://github.com/SoftRoCE/librxe-dev librxe - source code in
Github
[2] https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home - Soft RoCE
Wiki page
[3] https://github.com/SoftRoCE/librxe-dev

 - Soft RoCE userspace library

Signed-off-by: default avatarKamal Heib <kamalh@mellanox.com>
Signed-off-by: default avatarAmir Vadai <amirv@mellanox.com>
Signed-off-by: default avatarMoni Shoua <monis@mellanox.com>
Reviewed-by: default avatarHaggai Eran <haggaie@mellanox.com>
Signed-off-by: default avatarDoug Ledford <dledford@redhat.com>
parent 33688abb

MAINTAINERS

+9 −0
Original line number Diff line number Diff line
@@ -7444,6 +7444,15 @@ W: http://www.mellanox.com 
Q:	http://patchwork.ozlabs.org/project/netdev/list/

F:	drivers/net/ethernet/mellanox/mlxsw/



SOFT-ROCE DRIVER (rxe)

M:	Moni Shoua <monis@mellanox.com>

L:	linux-rdma@vger.kernel.org

S:	Supported

W:	https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home

Q:	http://patchwork.kernel.org/project/linux-rdma/list/

F:	drivers/infiniband/hw/rxe/

F:	include/uapi/rdma/rdma_user_rxe.h



MEMBARRIER SUPPORT

M:	Mathieu Desnoyers <mathieu.desnoyers@efficios.com>

M:	"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>

drivers/infiniband/Kconfig

+1 −0
Original line number Diff line number Diff line
@@ -84,6 +84,7 @@ source "drivers/infiniband/ulp/iser/Kconfig" 
source "drivers/infiniband/ulp/isert/Kconfig"



source "drivers/infiniband/sw/rdmavt/Kconfig"

source "drivers/infiniband/sw/rxe/Kconfig"



source "drivers/infiniband/hw/hfi1/Kconfig"

drivers/infiniband/sw/Makefile

+1 −0
Original line number Diff line number Diff line 
obj-$(CONFIG_INFINIBAND_RDMAVT)		+= rdmavt/

obj-$(CONFIG_RDMA_RXE)			+= rxe/

drivers/infiniband/sw/rxe/Kconfig

0 → 100644
+24 −0
Original line number Diff line number Diff line 
config RDMA_RXE

	tristate "Software RDMA over Ethernet (RoCE) driver"

	depends on INET && PCI && INFINIBAND

	depends on NET_UDP_TUNNEL

	---help---

	This driver implements the InfiniBand RDMA transport over

	the Linux network stack. It enables a system with a

	standard Ethernet adapter to interoperate with a RoCE

	adapter or with another system running the RXE driver.

	Documentation on InfiniBand and RoCE can be downloaded at

	www.infinibandta.org and www.openfabrics.org. (See also

	siw which is a similar software driver for iWARP.)



	The driver is split into two layers, one interfaces with the

	Linux RDMA stack and implements a kernel or user space

	verbs API. The user space verbs API requires a support

	library named librxe which is loaded by the generic user

	space verbs API, libibverbs. The other layer interfaces

	with the Linux network stack at layer 3.



	To configure and work with soft-RoCE driver please use the

	following wiki page under "configure Soft-RoCE (RXE)" section:



	https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home

drivers/infiniband/sw/rxe/Makefile

0 → 100644
+24 −0
Original line number Diff line number Diff line 
obj-$(CONFIG_RDMA_RXE) += rdma_rxe.o



rdma_rxe-y := \

	rxe.o \

	rxe_comp.o \

	rxe_req.o \

	rxe_resp.o \

	rxe_recv.o \

	rxe_pool.o \

	rxe_queue.o \

	rxe_verbs.o \

	rxe_av.o \

	rxe_srq.o \

	rxe_qp.o \

	rxe_cq.o \

	rxe_mr.o \

	rxe_dma.o \

	rxe_opcode.o \

	rxe_mmap.o \

	rxe_icrc.o \

	rxe_mcast.o \

	rxe_task.o \

	rxe_net.o \

	rxe_sysfs.o