[media] rc: make s_tx_mask consistent

When s_tx_mask is given an invalid bitmask, the number of transmitters
should be returned. See the LIRC_SET_TRANSMITTER_MASK lirc ioctl
documentation.

Signed-off-by: Sean Young <sean@mess.org>
Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>