modsign: Make sign-file determine the format of the X.509 cert

Make sign-file determine the format of the X.509 certificate by reading the
first two bytes and seeing if the first byte is 0x30 and the second
0x81-0x84.  If this is the case, assume it's DER encoded, otherwise assume
it to be PEM encoded.

Without this, it gets awkward to deal with the error messages from
d2i_X509_bio() when we want to call BIO_reset() and then PEM_read_bio() in
case the certificate was PEM encoded rather than X.509 encoded.

Reported-by: Ben Hutchings <ben@decadent.org.uk>
Signed-off-by: David Howells <dhowells@redhat.com>
Tested-by: Ben Hutchings <ben@decadent.org.uk>
cc: David Woodhouse <dwmw2@infradead.org>
cc: Juerg Haefliger <juerg.haefliger@hpe.com>
cc: Ben Hutchings <ben@decadent.org.uk>