Remove unnecessay icons on QuickContact Recent card

Bug: 28944282
Test: Manually verified icons are removed and the rest UI stays the same
when:
1. recent card has no expand button
2. recent card is expanded/collapsed in portrait, landscape, RTL

Change-Id: I550eb75b716f0ef15d8f243c4c1f0a57d726a3e5