Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit ee7cd898 authored by Rusty Russell's avatar Rusty Russell
Browse files

virtio: expose added descriptors immediately. 



A virtio driver does virtqueue_add_buf() multiple times before finally
calling virtqueue_kick(); previously we only exposed the added buffers
in the virtqueue_kick() call.  This means we don't need a memory
barrier in virtqueue_add_buf(), but it reduces concurrency as the
device (ie. host) can't see the buffers until the kick.

In the unusual (but now possible) case where a driver does add_buf()
and get_buf() without doing a kick, we do need to insert one before
our counter wraps.  Otherwise we could wrap num_added, and later on
not realize that we have passed the marker where we should have
kicked.

Signed-off-by: default avatarRusty Russell <rusty@rustcorp.com.au>
parent 3b720b8c

drivers/virtio/virtio_ring.c

+14 −6
Original line number Original line Diff line number Diff line
@@ -251,9 +251,20 @@ int virtqueue_add_buf(struct virtqueue *_vq, 




	/* Put entry in available array (but don't update avail->idx until they

	/* Put entry in available array (but don't update avail->idx until they

	 * do sync). */

	 * do sync). */

	avail = ((vq->vring.avail->idx + vq->num_added++) & (vq->vring.num-1));

	avail = (vq->vring.avail->idx & (vq->vring.num-1));

	vq->vring.avail->ring[avail] = head;

	vq->vring.avail->ring[avail] = head;





	/* Descriptors and available array need to be set before we expose the

	 * new available array entries. */

	virtio_wmb(vq);

	vq->vring.avail->idx++;

	vq->num_added++;



	/* This is very unlikely, but theoretically possible.  Kick

	 * just in case. */

	if (unlikely(vq->num_added == (1 << 16) - 1))

		virtqueue_kick(_vq);



	pr_debug("Added buffer head %i to %p\n", head, vq);

	pr_debug("Added buffer head %i to %p\n", head, vq);

	END_USE(vq);

	END_USE(vq);
@@ -283,13 +294,10 @@ bool virtqueue_kick_prepare(struct virtqueue *_vq) 
	 * new available array entries. */

	 * new available array entries. */

	virtio_wmb(vq);

	virtio_wmb(vq);





	old = vq->vring.avail->idx;

	old = vq->vring.avail->idx - vq->num_added;

	new = vq->vring.avail->idx = old + vq->num_added;

	new = vq->vring.avail->idx;

	vq->num_added = 0;

	vq->num_added = 0;





	/* Need to update avail index before checking if we should notify */

	virtio_mb(vq);



	if (vq->event) {

	if (vq->event) {

		needs_kick = vring_need_event(vring_avail_event(&vq->vring),

		needs_kick = vring_need_event(vring_avail_event(&vq->vring),

					      new, old);

					      new, old);