hal: remove sending apply audio path for speaker swap

For speaker swap channels, apply audio path  need not be
sent when swap is not requested or device switch is not
needed. Sending apply audio path will increment the ref
count in audio route which can lead to erroneous condition.

Change-Id: I64e3a867cd749c56d3432f1443f2cc4a92894374