slim-msm: Handling resources during port de-allocation

In port de-allocate operation free the resources
only if port is connected as memory allocates while
connecting the port.

Change-Id: I743a27025e83f0f4bb8a47c20b2bb2d4cff80caa
Signed-off-by: Dilip Kota <dkota@codeaurora.org>