Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux

  https://en.wikipedia.org/wiki/Channel_I/O

- s390 architecture:

  s390 Principles of Operation manual (IBM Form. No. SA22-7832)

- The existing Qemu code which implements a simple emulated channel

- The existing QEMU code which implements a simple emulated channel

  subsystem could also be a good reference. It makes it easier to follow

  the flow.

  qemu/hw/s390x/css.c

Motivation of vfio-ccw

----------------------

Currently, a guest virtualized via qemu/kvm on s390 only sees

Typically, a guest virtualized via QEMU/KVM on s390 only sees

paravirtualized virtio devices via the "Virtio Over Channel I/O

(virtio-ccw)" transport. This makes virtio devices discoverable via

standard operating system algorithms for handling channel devices.

However this is not enough. On s390 for the majority of devices, which

use the standard Channel I/O based mechanism, we also need to provide

the functionality of passing through them to a Qemu virtual machine.

the functionality of passing through them to a QEMU virtual machine.

This includes devices that don't have a virtio counterpart (e.g. tape

drives) or that have specific characteristics which guests want to

exploit.

For passing a device to a guest, we want to use the same interface as

everybody else, namely vfio. Thus, we would like to introduce vfio

support for channel devices. And we would like to name this new vfio

device "vfio-ccw".

everybody else, namely vfio. We implement this vfio support for channel

devices via the vfio mediated device framework and the subchannel device

driver "vfio_ccw".

Access patterns of CCW devices

------------------------------

Channel I/O does not have IOMMU hardware support, so the physical

vfio-ccw device does not have an IOMMU level translation or isolation.

Sub-channel I/O instructions are all privileged instructions, When

Subchannel I/O instructions are all privileged instructions. When

handling the I/O instruction interception, vfio-ccw has the software

policing and translation how the channel program is programmed before

it gets sent to hardware.

- The vfio_mdev driver for the mediated vfio ccw device.

  This is provided by the mdev framework. It is a vfio device driver for

  the mdev that created by vfio_ccw.

  It realize a group of vfio device driver callbacks, adds itself to a

  It realizes a group of vfio device driver callbacks, adds itself to a

  vfio group, and registers itself to the mdev framework as a mdev

  driver.

  It uses a vfio iommu backend that uses the existing map and unmap

An I/O region is used to accept channel program request from user

space and store I/O interrupt result for user space to retrieve. The

defination of the region is:

definition of the region is:

struct ccw_io_region {

#define ORB_AREA_SIZE 12

ret_code stores a return code for each access of the region.

vfio-ccw patches overview

-------------------------

vfio-ccw operation details

--------------------------

For now, our patches are rebased on the latest mdev implementation.

vfio-ccw follows what vfio-pci did on the s390 paltform and uses

vfio-iommu-type1 as the vfio iommu backend. It's a good start to launch

the code review for vfio-ccw. Note that the implementation is far from

complete yet; but we'd like to get feedback for the general

architecture.

vfio-ccw follows what vfio-pci did on the s390 platform and uses

vfio-iommu-type1 as the vfio iommu backend.

* CCW translation APIs

- Description:

  These introduce a group of APIs (start with 'cp_') to do CCW

  translation. The CCWs passed in by a user space program are

  organized with their guest physical memory addresses. These APIs

  will copy the CCWs into the kernel space, and assemble a runnable

  kernel channel program by updating the guest physical addresses with

  their corresponding host physical addresses.

- Patches:

  vfio: ccw: introduce channel program interfaces

  A group of APIs (start with 'cp_') to do CCW translation. The CCWs

  passed in by a user space program are organized with their guest

  physical memory addresses. These APIs will copy the CCWs into kernel

  space, and assemble a runnable kernel channel program by updating the

  guest physical addresses with their corresponding host physical addresses.

  Note that we have to use IDALs even for direct-access CCWs, as the

  referenced memory can be located anywhere, including above 2G.

* vfio_ccw device driver

- Description:

  The following patches utilizes the CCW translation APIs and introduce

  This driver utilizes the CCW translation APIs and introduces

  vfio_ccw, which is the driver for the I/O subchannel devices you want

  to pass through.

  vfio_ccw implements the following vfio ioctls:

  This also provides the SET_IRQ ioctl to setup an event notifier to

  notify the user space program the I/O completion in an asynchronous

  way.

- Patches:

  vfio: ccw: basic implementation for vfio_ccw driver

  vfio: ccw: introduce ccw_io_region

  vfio: ccw: realize VFIO_DEVICE_GET_REGION_INFO ioctl

  vfio: ccw: realize VFIO_DEVICE_RESET ioctl

  vfio: ccw: realize VFIO_DEVICE_G(S)ET_IRQ_INFO ioctls

The user of vfio-ccw is not limited to Qemu, while Qemu is definitely a

The use of vfio-ccw is not limited to QEMU, while QEMU is definitely a

good example to get understand how these patches work. Here is a little

bit more detail how an I/O request triggered by the Qemu guest will be

bit more detail how an I/O request triggered by the QEMU guest will be

handled (without error handling).

Explanation:

Q1-Q7: Qemu side process.

Q1-Q7: QEMU side process.

K1-K5: Kernel side process.

Q1. Get I/O region info during initialization.

    K2. Translate the guest channel program to a host kernel space

        channel program, which becomes runnable for a real device.

    K3. With the necessary information contained in the orb passed in

        by Qemu, issue the ccwchain to the device.

        by QEMU, issue the ccwchain to the device.

    K4. Return the ssch CC code.

Q5. Return the CC code to the guest.

    K5. Interrupt handler gets the I/O result and write the result to

        the I/O region.

    K6. Signal Qemu to retrieve the result.

    K6. Signal QEMU to retrieve the result.

Q6. Get the signal and event handler reads out the result from the I/O

    region.

Q7. Update the irb for the guest.

https://en.wikipedia.org/wiki/Direct-access_storage_device

https://en.wikipedia.org/wiki/Count_key_data

Together with the corresponding work in Qemu, we can bring the passed

Together with the corresponding work in QEMU, we can bring the passed

through DASD/ECKD device online in a guest now and use it as a block

device.

While the current code allows the guest to start channel programs via

START SUBCHANNEL, support for HALT SUBCHANNEL or CLEAR SUBCHANNEL is

not yet implemented.

vfio-ccw supports classic (command mode) channel I/O only. Transport

mode (HPF) is not supported.

QDIO subchannels are currently not supported. Classic devices other than

DASD/ECKD might work, but have not been tested.

Reference

---------

1. ESA/s390 Principles of Operation manual (IBM Form. No. SA22-7832)

	select GENERIC_CLOCKEVENTS

	select GENERIC_CPU_AUTOPROBE

	select GENERIC_CPU_DEVICES if !SMP

	select GENERIC_CPU_VULNERABILITIES

	select GENERIC_FIND_FIRST_BIT

	select GENERIC_SMP_IDLE_THREAD

	select GENERIC_TIME_VSYSCALL

config EXPOLINE_OFF

	bool "spectre_v2=off"

config EXPOLINE_MEDIUM

config EXPOLINE_AUTO

	bool "spectre_v2=auto"

config EXPOLINE_FULL

cflags-y += -Wa,-I$(srctree)/arch/$(ARCH)/include

#KBUILD_IMAGE is necessary for make rpm

KBUILD_IMAGE	:=arch/s390/boot/image

#

# Prevent tail-call optimizations, to get clearer backtraces:

#

    CC_FLAGS_EXPOLINE += -mfunction-return=thunk

    CC_FLAGS_EXPOLINE += -mindirect-branch-table

    export CC_FLAGS_EXPOLINE

    cflags-y += $(CC_FLAGS_EXPOLINE)

    cflags-y += $(CC_FLAGS_EXPOLINE) -DCC_USING_EXPOLINE

  endif

endif

all: image bzImage

#KBUILD_IMAGE is necessary for packaging targets like rpm-pkg, deb-pkg...

KBUILD_IMAGE	:= $(boot)/bzImage

install: vmlinux

	$(Q)$(MAKE) $(build)=$(boot) $@

$(obj)/vmlinux: $(obj)/vmlinux.lds $(OBJECTS)

	$(call if_changed,ld)

sed-sizes := -e 's/^\([0-9a-fA-F]*\) . \(__bss_start\|_end\)$$/\#define SZ\2 0x\1/p'

TRIM_HEAD_SIZE := 0x11000

sed-sizes := -e 's/^\([0-9a-fA-F]*\) . \(__bss_start\|_end\)$$/\#define SZ\2 (0x\1 - $(TRIM_HEAD_SIZE))/p'

quiet_cmd_sizes = GEN     $@

      cmd_sizes = $(NM) $< | sed -n $(sed-sizes) > $@

quiet_cmd_trim_head = TRIM    $@

      cmd_trim_head = tail -c +$$(($(TRIM_HEAD_SIZE) + 1)) $< > $@

$(obj)/sizes.h: vmlinux

	$(call if_changed,sizes)

CFLAGS_misc.o += -I$(objtree)/$(obj)

$(obj)/misc.o: $(obj)/sizes.h

OBJCOPYFLAGS_vmlinux.bin :=  -R .comment -S

$(obj)/vmlinux.bin: vmlinux

OBJCOPYFLAGS_vmlinux.bin.full :=  -R .comment -S

$(obj)/vmlinux.bin.full: vmlinux

	$(call if_changed,objcopy)

$(obj)/vmlinux.bin: $(obj)/vmlinux.bin.full

	$(call if_changed,trim_head)

vmlinux.bin.all-y := $(obj)/vmlinux.bin

suffix-$(CONFIG_KERNEL_GZIP)  := gz

	aghi	%r15,-160

	brasl	%r14,decompress_kernel

	# Set up registers for memory mover. We move the decompressed image to

	# 0x11000, starting at offset 0x11000 in the decompressed image so

	# that code living at 0x11000 in the image will end up at 0x11000 in

	# memory.

	# 0x11000, where startup_continue of the decompressed image is supposed

	# to be.

	lgr	%r4,%r2

	lg	%r2,.Loffset-.LPG1(%r13)

	la	%r4,0(%r2,%r4)

	lg	%r3,.Lmvsize-.LPG1(%r13)

	lgr	%r5,%r3

	# Move the memory mover someplace safe so it doesn't overwrite itself.