V4L/DVB (5678): Zr364xx: fix return values

This patch fixes several return value related problems in zr364xx.

- return -ENOMEM instead of -ENODEV on out of memory

- zr364xx checks video_register_device() error only when
  its return value is -1. But video_register_device() doesn't
  always return -1 on error.

- If usb_register() returns error, module_init() wrongly returns 1:
	retval = usb_register(&zr364xx_driver) < 0;
	...
	return retval;

  And it allows the module to be loaded. Because sys_init_module() doesn't
  see positive return value as error.

Signed-off-by: Akinobu Mita <akinobu.mita@gmail.com>
Signed-off-by: Antoine Jacquet <royale@zerezo.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab@infradead.org>