video: driver: Return proper error code

Return correct error code when session is not
supported and when max memory limit is
reached.

Change-Id: I5ed3770ff4333b8828accbbb11b740ca316440b5
Signed-off-by: Uma Mehta <quic_umamehta@quicinc.com>