netfilter: x_tables: output bad hook mask in hexadecimal

It is a mask, and masks are most useful in hex.

Signed-off-by: Jan Engelhardt <jengelh@medozas.de>
Signed-off-by: Patrick McHardy <kaber@trash.net>