NFSv4: Fix up delegation callbacks

Currently, the callback server is listening on IPv6 if it is enabled. This
means that IPv4 addresses will always be mapped.

Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>