Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 28dbabef authored by Eric Dumazet's avatar Eric Dumazet Committed by Greg Kroah-Hartman
Browse files

rfs: annotate lockless accesses to RFS sock flow table 



[ Upstream commit 5c3b74a92aa285a3df722bf6329ba7ccf70346d6 ]

Add READ_ONCE()/WRITE_ONCE() on accesses to the sock flow table.

This also prevents a (smart ?) compiler to remove the condition in:

if (table->ents[index] != newval)
        table->ents[index] = newval;

We need the condition to avoid dirtying a shared cache line.

Fixes: fec5e652 ("rfs: Receive Flow Steering")
Signed-off-by: default avatarEric Dumazet <edumazet@google.com>
Reviewed-by: default avatarSimon Horman <simon.horman@corigine.com>
Reviewed-by: default avatarKuniyuki Iwashima <kuniyu@amazon.com>
Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
Signed-off-by: default avatarSasha Levin <sashal@kernel.org>
parent 9b3907c5

include/linux/netdevice.h

+5 −2
Original line number Diff line number Diff line
@@ -695,8 +695,11 @@ static inline void rps_record_sock_flow(struct rps_sock_flow_table *table, 
		/* We only give a hint, preemption can change CPU under us */

		val |= raw_smp_processor_id();



		if (table->ents[index] != val)

			table->ents[index] = val;

		/* The following WRITE_ONCE() is paired with the READ_ONCE()

		 * here, and another one in get_rps_cpu().

		 */

		if (READ_ONCE(table->ents[index]) != val)

			WRITE_ONCE(table->ents[index], val);

	}

}

net/core/dev.c

+4 −2
Original line number Diff line number Diff line
@@ -4064,8 +4064,10 @@ static int get_rps_cpu(struct net_device *dev, struct sk_buff *skb, 
		u32 next_cpu;

		u32 ident;



		/* First check into global flow table if there is a match */

		ident = sock_flow_table->ents[hash & sock_flow_table->mask];

		/* First check into global flow table if there is a match.

		 * This READ_ONCE() pairs with WRITE_ONCE() from rps_record_sock_flow().

		 */

		ident = READ_ONCE(sock_flow_table->ents[hash & sock_flow_table->mask]);

		if ((ident ^ hash) & ~rps_cpu_mask)

			goto try_rps;