[media] r820t: fix PLL calculus

There are a few errors at the PLL calculus, causing the device
to use wrong values.
While here, change the calculus to use 32 bits, as there's no
need for 64 bits there.

Signed-off-by: Mauro Carvalho Chehab <mchehab@redhat.com>
Tested-by: Antti Palosaari <crope@iki.fi>