Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 1077fa36 authored by Alexander Duyck's avatar Alexander Duyck Committed by David S. Miller
Browse files

arch: Add lightweight memory barriers dma_rmb() and dma_wmb() 



There are a number of situations where the mandatory barriers rmb() and
wmb() are used to order memory/memory operations in the device drivers
and those barriers are much heavier than they actually need to be.  For
example in the case of PowerPC wmb() calls the heavy-weight sync
instruction when for coherent memory operations all that is really needed
is an lsync or eieio instruction.

This commit adds a coherent only version of the mandatory memory barriers
rmb() and wmb().  In most cases this should result in the barrier being the
same as the SMP barriers for the SMP case, however in some cases we use a
barrier that is somewhere in between rmb() and smp_rmb().  For example on
ARM the rmb barriers break down as follows:

  Barrier   Call     Explanation
  --------- -------- ----------------------------------
  rmb()     dsb()    Data synchronization barrier - system
  dma_rmb() dmb(osh) data memory barrier - outer sharable
  smp_rmb() dmb(ish) data memory barrier - inner sharable

These new barriers are not as safe as the standard rmb() and wmb().
Specifically they do not guarantee ordering between coherent and incoherent
memories.  The primary use case for these would be to enforce ordering of
reads and writes when accessing coherent memory that is shared between the
CPU and a device.

It may also be noted that there is no dma_mb().  Most architectures don't
provide a good mechanism for performing a coherent only full barrier without
resorting to the same mechanism used in mb().  As such there isn't much to
be gained in trying to define such a function.

Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
Cc: Michael Ellerman <michael@ellerman.id.au>
Cc: Michael Neuling <mikey@neuling.org>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: David Miller <davem@davemloft.net>
Acked-by: default avatarBenjamin Herrenschmidt <benh@kernel.crashing.org>
Acked-by: default avatarWill Deacon <will.deacon@arm.com>
Signed-off-by: default avatarAlexander Duyck <alexander.h.duyck@redhat.com>
Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
parent 8a449718

Documentation/memory-barriers.txt

+42 −0
Original line number Diff line number Diff line
@@ -1633,6 +1633,48 @@ There are some more advanced barrier functions: 
     operations" subsection for information on where to use these.





 (*) dma_wmb();

 (*) dma_rmb();



     These are for use with consistent memory to guarantee the ordering

     of writes or reads of shared memory accessible to both the CPU and a

     DMA capable device.



     For example, consider a device driver that shares memory with a device

     and uses a descriptor status value to indicate if the descriptor belongs

     to the device or the CPU, and a doorbell to notify it when new

     descriptors are available:



	if (desc->status != DEVICE_OWN) {

		/* do not read data until we own descriptor */

		dma_rmb();



		/* read/modify data */

		read_data = desc->data;

		desc->data = write_data;



		/* flush modifications before status update */

		dma_wmb();



		/* assign ownership */

		desc->status = DEVICE_OWN;



		/* force memory to sync before notifying device via MMIO */

		wmb();



		/* notify device of new descriptors */

		writel(DESC_NOTIFY, doorbell);

	}



     The dma_rmb() allows us guarantee the device has released ownership

     before we read the data from the descriptor, and he dma_wmb() allows

     us to guarantee the data is written to the descriptor before the device

     can see it now has ownership.  The wmb() is needed to guarantee that the

     cache coherent memory writes have completed before attempting a write to

     the cache incoherent MMIO region.



     See Documentation/DMA-API.txt for more information on consistent memory.



MMIO WRITE BARRIER

------------------

arch/arm/include/asm/barrier.h

+4 −0
Original line number Diff line number Diff line
@@ -43,10 +43,14 @@ 
#define mb()		do { dsb(); outer_sync(); } while (0)

#define rmb()		dsb()

#define wmb()		do { dsb(st); outer_sync(); } while (0)

#define dma_rmb()	dmb(osh)

#define dma_wmb()	dmb(oshst)

#else

#define mb()		barrier()

#define rmb()		barrier()

#define wmb()		barrier()

#define dma_rmb()	barrier()

#define dma_wmb()	barrier()

#endif



#ifndef CONFIG_SMP

arch/arm64/include/asm/barrier.h

+3 −0
Original line number Diff line number Diff line
@@ -32,6 +32,9 @@ 
#define rmb()		dsb(ld)

#define wmb()		dsb(st)



#define dma_rmb()	dmb(oshld)

#define dma_wmb()	dmb(oshst)



#ifndef CONFIG_SMP

#define smp_mb()	barrier()

#define smp_rmb()	barrier()

arch/ia64/include/asm/barrier.h

+3 −0
Original line number Diff line number Diff line
@@ -39,6 +39,9 @@ 
#define rmb()		mb()

#define wmb()		mb()



#define dma_rmb()	mb()

#define dma_wmb()	mb()



#ifdef CONFIG_SMP

# define smp_mb()	mb()

#else

arch/metag/include/asm/barrier.h

+7 −7
Original line number Diff line number Diff line
@@ -4,8 +4,6 @@ 
#include <asm/metag_mem.h>



#define nop()		asm volatile ("NOP")

#define mb()		wmb()

#define rmb()		barrier()



#ifdef CONFIG_METAG_META21
@@ -41,11 +39,13 @@ static inline void wr_fence(void) 


#endif /* !CONFIG_METAG_META21 */



static inline void wmb(void)

{

/* flush writes through the write combiner */

	wr_fence();

}

#define mb()		wr_fence()

#define rmb()		barrier()

#define wmb()		mb()



#define dma_rmb()	rmb()

#define dma_wmb()	wmb()



#ifndef CONFIG_SMP

#define fence()		do { } while (0)