broadcom: Fix slow link problem

When a 50610 or 50610M is paired against particular remote partners,
link is slow to come up.  This patch works around the problem.

Signed-off-by: Matt Carlson <mcarlson@broadcom.com>
Reviewed-by: Michael Chan <mchan@broadcom.com>
Signed-off-by: David S. Miller <davem@davemloft.net>