drm/nva3/clk: minor improvements to fractional N calculation

Helps us to get identical numbers to the binary driver for (at least)
Kepler memory PLLs, and fixes a rounding error.

Signed-off-by: Ben Skeggs <bskeggs@redhat.com>