[media] mem2mem_testdev: unlock and return error code properly

We recently added locking to this function, but there was an error path
which accidentally returned holding a lock.  Also we returned zero on
failure on some paths instead of the error code.

Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab@redhat.com>