net: ipc_router: Fix remote port conn_info memory leak

conn_info is used to connect the local port with destination port
address and is freed only for remote server cleanup in SSR case
which leads memory leak for other remote ports which valid conn_info.

Free the conn_info structure for all remote ports during the SSR and
while handling remove client message.

CRs-Fixed: 1057219
Change-Id: I164a9eb308f5779d545766b18bb41184c49bbb3d
Signed-off-by: Arun Kumar Neelakantam <aneela@codeaurora.org>