RTSP: Correct the data type in the check

The check for a race condition in RTSP flush has a mismatch
in its data type.
UNKNOWN_ERROR is used as a positive value and isn't equal to
the return value of feedMoreTSData().
Typecast it to fix this mismatch.

Change-Id: Ib8a88af13e5b43f5802052cb8a956c2f391e73b3