ColorConverter: fix handling unspecified range (and one matrix coeff)

We default to limited range for matrix selection, but coefficient selection
defaulted to full range which resulted in a calculation error and off colors.

Also fixed a typo on the 10-bit BT.709 matrix resulting in a slight color
shift.

Bug: 287750721
Test: POC in bug descriptions

Change-Id: I5dcc1a87331ea24ccfcdfbeadd244e2acb464677