V4L/DVB (5412): Fix VIDIOC_TRY_ENCODER_CMD and VIDEO_TRY_COMMAND

VIDIOC_TRY_ENCODER_CMD did the same as VIDIOC_ENCODER_CMD, now it no longer
touches the encoder.
Both the encoder and decoder commands did not clear the flags field of unknown
flags.

Signed-off-by: Hans Verkuil <hverkuil@xs4all.nl>
Signed-off-by: Mauro Carvalho Chehab <mchehab@infradead.org>